System and method for collaborative shopping, business and entertainment

ABSTRACT

The methods and systems described herein relate to online methods of collaboration in community environments. The methods and systems are related to an online apparel modeling system that allows users to have three-dimensional models of their physical profile created. Users may purchase various goods and/or services and collaborate with other users in the online environment.

This application claims the benefit of Provisional Application No.61/064,716, filed Mar. 21, 2008, which is hereby incorporated herein byreference.

FIELD

The embodiments described herein relate generally to immersive onlineshopping, entertainment, business, travel and product modeling, inparticular to a method and system for modeling of apparel items onlinein a collaborative environment.

BACKGROUND

Times have changed. There has been a dramatic rise in nuclear familiesand this coupled with increasing globalization is affecting the way welive, work, and interact. But humans will continue to remain human; thehuman instinct to form communities, stay connected, interact andcollaborate still exists. There is a need to facilitate and ease theseprocesses in a new era of ever-growing population and information wheretime is precious. The experience of real face-to-face interaction isoften missing. Technology has to emulate components of real experiencesand human factors in order for users to be fully satisfied.

An ever growing segment of the population is relying on the Internet topurchase various products and services. Offerings such as those relatedto travel have become ever more popular with respect to onlinepurchasing. As users are generally familiar with their travelrequirements, and adequate information is provided online for users tomake their travel decisions, many users make all of their travelbookings online.

While there has been an increase in the percentage of people purchasingitems of apparel online, it has not mirrored the percentages of peoplethat purchase goods and services such as travel packages online. One ofthe main reasons for the different rates of adoption is because of therequirements associated with purchasing items of apparel. One of themain requirements when purchasing apparel whether purchased online orthrough a conventional establishment is to ensure that the item fits.The determination of whether an item fits often cannot be made withregards to just the displayed or stated size of the item. Items fromdifferent manufacturers though of the same size, often fit differently.Therefore, people often wish to be able to try on the items beforepurchasing to determine the suitability of fit, and how it appears.

Further, when shopping for items of apparel, people generally enjoy thesocial components of shopping. Many people will often take others tostores when purchasing apparel for the feedback or even company. As aresult of the limitations associated with current models for onlineapparel shopping, the public has not been as ready to adopt suchshopping methods. Methods are needed to facilitate collaboration anddecision making, and for emulating reality through technology in allfacets of the user's life including work, business, study, research,travel, legal affairs, family life, entertainment, and shopping.

SUMMARY

The methods and systems described herein relate to online methods ofcollaboration in community environments. The methods and systems arerelated to an online apparel modeling system that allows users to havethree-dimensional models of their physical profile created. Users maypurchase various goods and/or services and collaborate with other usersin the online environment.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the embodiments described herein and toshow more clearly how they may be carried into effect, reference willnow be made, by way of example only, to the accompanying drawings whichshow at least one exemplary embodiment, and in which:

FIG. 1 is a block diagram of the components of a shopping,entertainment, and business system;

FIG. 2 is a block diagram of the components of a computing device;

FIG. 3 is a block diagram of the components of a server application;

FIG. 4 is a block diagram of the components of a data store;

FIG. 5 is a flowchart diagram of an access method;

FIG. 6A-J illustrate the model generation method;

FIG. 7A-D illustrate the modes of operation in a collaborativeenvironment;

FIG. 8 is an image of a sample main page screen for shopping;

FIG. 9 is an image of a sample upload window for data for modelgeneration;

FIG. 10 is a image of a sample local application window and a samplebrowser window;

FIG. 11 is an image of a sample facial synthesis window;

FIG. 12A is an image of a sample measurement window;

FIG. 12B is an image of a sample constructed photorealistic model;

FIG. 12C is another image of a sample constructed photorealistic model;

FIG. 13A is an image of a set of non photorealistic renderings of theuser model shown from different viewpoints;

FIG. 13B is an image showing a sample mechanism that allows users tomake body modifications directly on the user model using hotspotregions;

FIG. 13C is an image showing a sample ruler for taking measurements ofthe user model;

FIG. 14 is an image of a sample environment manager;

FIG. 15A is an image of a sample user model environment;

FIG. 15B is an image illustrating sample features of collaborativeshopping;

FIG. 16 is a sample image of a component of a Shopping Trip managementpanel;

FIG. 17 is an image of a sample friends manager window;

FIG. 18 is an image of a sample friendship management window;

FIG. 19 is an image of a sample chat window;

FIG. 20 is an image of a sample collaborative environment;

FIG. 21A-G are images illustrating Split-Bill features;

FIG. 22 is an image of a sample apparel display window;

FIG. 23 is an image of a shared item window;

FIG. 24 is an image of a sample fitting room window in a browser window;

FIG. 25 is an image of a sample wardrobe item;

FIG. 26 is an image of a sample wardrobe consultant window;

FIG. 27 is an image describing a sample instance of user interactionwith the wardrobe and fitting room;

FIG. 28 is an image of a sample 3D realization of a virtual wardrobe;

FIG. 29A is an image showing sample visual sequences displayed to a userwhile the apparel and hair is being modeled and fitted on the usermodel.

FIG. 29B is an image illustrating sample mechanisms available to theuser for making body adjustments to their user model;

FIG. 29C is an image showing sample product catalogue views available tothe user and a sample mechanism for trying on a product in the catalogueon the user model;

FIG. 30 is an image showing sample visualization schemes for fitinformation with respect to the body surface;

FIG. 31 is an image of a sample browser main page screen and a samplelocal application screen, showing sample features;

FIG. 32 is an image of a sample user model environment;

FIG. 33 is an image of a sample user model environment with samplevirtual components;

FIG. 34 is an image where a sample user model music video is shown;

FIG. 35 is an image showing sample manipulations of a user model'sexpressions and looks;

FIG. 36 is an image of a sample virtual store window showing virtualinteraction between a user and a sales service representative;

FIG. 37 is an outline of a sample ADF file in XML format;

FIG. 38 is a flowchart diagram that provides an overview of ADF filecreation and use;

FIG. 39A is in image of a sample procedure for a user to gain access tofriends on system 10 from the user's account on a social networking sitesuch as Facebook;

FIG. 39B is an image of a sample user account page on system 10 before auser has logged into Facebook;

FIG. 39C is an image of a sample page for accessing a social networkingsite (Facebook) through system 10;

FIG. 39D is an image of a sample user account page on system 10 after auser has logged into Facebook;

FIG. 40 is a sample image of a Shopping Trip management panel;

FIG. 41A-F are snapshots of a sample realization of the system discussedwith reference to FIG. 20;

FIG. 42 illustrates a sample interaction between various parties usingsystem 10;

FIG. 43 is an image illustrating sample features of the hangout zone;

FIG. 44 is an image of a sample main page in the hangout zone;

FIG. 45 is an image of a sample style browser display window;

FIG. 46A is an image of another sample main page for shopping;

FIG. 46B is an image of a sample store window;

FIG. 46C is an image of another sample store window;

FIG. 46D is an image of sample shopping trip window;

FIG. 46E is an image of a user's sample personalized looks window;

FIG. 46F is an image of a sample fitting room window;

FIG. 46G is an image of another sample fitting room window;

FIG. 46H is an image of a sample shopping diary window;

FIG. 46I is an image of a sample directory page;

FIG. 47A-B are sample images illustrating a feature that allows users tocustomize the look and feel of the browser application;

FIGS. 48A-F, are images illustrating sample layout designs and selectfeatures of system 10;

FIGS. 49A-O are images illustrating sample features of the AFMS/VOS;

FIG. 49L is an image of the sample storage structure of the AFMS/VOS;

FIG. 49M is an image of a sample user accounts management structurewithin the AFMS/VOS;

FIG. 49N is an image that shows sample abstraction of a search querythat is fed into the search engine that is a part of the AFMS/VOS;

FIG. 49O is an image of a sample implementation of the AFMS/VOS as awebsite;

FIG. 49P is an image of a sample application management structure withinthe AFMS/VOS;

FIG. 49Q is an image of an exemplary embodiment of file tagging,sharing, and searching features in the VOS/AFMS;

FIG. 49R is a sample image of a user interface for filtering searchdata;

FIG. 49S is a sample image of an interface to the object oriented filesystem;

FIG. 50 illustrates a sample mobile communication system when a user isin a store;

FIG. 51A illustrates a sample communication network demonstratingexternal connections to system 10;

FIG. 51B illustrates a sample flowchart showing the operation of the VS;

FIG. 52A illustrates an image/video/audio analysis module for genericscene analysis;

FIG. 52B illustrates a method for detecting surprise;

FIG. 53 illustrates a sample interface for broadcasting andcollaborative communication;

FIG. 54A-F novel devices for human-computer interaction;

FIG. 55 illustrates an exemplary embodiment of a method foraudio/video/text summarization; and

FIG. 56 illustrates a sample usage of a collaborative VS application;

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION

It will be appreciated that, for simplicity and clarity of illustration,where considered appropriate, reference numerals may be repeated amongthe figures to indicate corresponding or analogous elements or steps. Inaddition, numerous specific details are set forth in order to provide athorough understanding of the exemplary embodiments described herein.However, it will be understood by those of ordinary skill in the artthat the embodiments described herein may be practiced without thesespecific details. In other instances, well-known methods, procedures andcomponents have not been described in detail so as not to obscure theembodiments described herein. Furthermore, this description is not to beconsidered as limiting the scope of the embodiments described herein inany way, but rather as merely describing the implementation of thevarious embodiments described herein.

The embodiments of the systems and methods described herein may beimplemented in hardware or software, or a combination of both. However,preferably, these embodiments are implemented in computer programsexecuting on programmable computers, each comprising at least oneprocessor, a data storage system (including volatile and non-volatilememory and/or storage elements), at least one input device, and at leastone output device. For example, and without limitation, the programmablecomputer may be a mainframe computer, server, personal computer, laptop,personal data assistant, or cellular telephone. A program code isapplied to input data to perform the functions described herein andgenerate output information. The output information is applied to one ormore output devices, in known fashion.

Each program is preferably implemented in a high level procedural orobject-oriented programming and/or scripting language to communicatewith a computer system. However, the programs can be implemented inassembly or machine language, if desired. In any case, the language maybe a compiled or interpreted language. Each such computer program ispreferably stored on a storage media or a device (e.g. ROM or magneticdiskette), readable by a general or special purpose programmablecomputer, for configuring and operating the computer when the storagemedia or device is read by the computer to perform the proceduresdescribed herein. The inventive system may also be considered to beimplemented as a computer-readable storage medium, configured with acomputer program, where the storage medium so configured causes acomputer to operate in a specific and predefined manner to perform thefunctions described herein.

Furthermore, the system, processes and methods of the describedembodiments are capable of being distributed in a computer programproduct comprising a computer-readable medium that bears computer-usableinstructions for one or more processors. The medium may be provided invarious forms, including one or more diskettes, compact disks, tapes,chips, wireline transmissions, satellite transmissions, internettransmissions or downloadings, magnetic and electronic storage media,digital and analog signals, and the like. The computer-usableinstructions may also be in various forms, including compiled andnon-compiled code.

Reference is now made to FIG. 1, wherein a block diagram illustratingcomponents of an online apparel modeling and collaboration system 10 areshown in an exemplary embodiment. The modeling system 10 allows users tohave three-dimensional models that are representative of their physicalprofile created. The three-dimensional models are herein referred to asuser models or character models, and are created based on informationprovided by the user. This information includes, but is not limited to,any combination of: images; movies; measurements; outlines of feet,hands, and other body parts; moulds/imprints including those of feet,hands, ears, and other body parts; scans such as laser scans; skin tone,race, gender, weight, hair type etc.; high resolution scans and imagesof the eyes; motion capture data (mocap). The users may then edit andmanipulate the user models that are created. The user models may then beused to model items of apparel. The virtual modeling of apparel providesthe user with an indication regarding the suitability of the apparel forthe user. The items of apparel may include, but are not limited to,items of clothing, jewelry, footwear, accessories, hair items, watches,and any other item that a user may adorn. The user is provided withvarious respective functionalities when using the system 10. Thefunctionalities include, but are not limited to, generating, viewing andediting three-dimensional models of users, viewing various apparel itemsplaced on the three-dimensional models, purchasing apparel items,interacting with other members of online communities, sharing thethree-dimensional models and sharing the apparel views with othermembers of the online communities. These features are representative of‘interactive shopping’ where users are not just limited to examiningdifferent views of a product before purchasing it from an electroniccatalogue but are able to examine 3D product simulations by putting themon their 3D virtual embodiments, interacting with products via theirvirtual model or directly, acquiring different perspectives of theproduct in 3D, getting acquainted with enhanced depictions of the lookand feel of the product as well as sharing all of these experiences andproduct manifestations with their social network. Media content thatcaptures the user model engaged in virtual activities such as game-play,singing, dancing, and other activities may also be shared. The usermodels may be exported to gaming environments including third partygames. The respective functionalities are described in further detailwith reference to FIGS. 2 to 50 Such a system may be generalized toinclude items other than apparel. In an exemplary embodiment, the usermay be presented with options for the color of a car that best matchesthe user's hair-color.

The online modeling system 10 in an exemplary embodiment comprises oneor more users 12 who interact with a respective computing device 14. Thecomputing devices 14 have resident upon them or associated with them aclient application 16 that may be used on the model generation processas described below. The respective computing devices 14 communicate witha portal server 20. The portal server 20 is implemented on a computingdevice and is used to control the operation of the system 10 and theuser's interaction with other members of the system 10 in an exemplaryembodiment. The portal server 20 has resident upon it or has associatedwith it a server application 22. The portal server 20 interacts withother servers that may be administered by third parties to providevarious functionalities to the user. In an exemplary embodiment, theonline modeling system 10 interacts with retail servers 24, communityservers 26, entertainment servers 23, media agency servers 25, financialinstitution servers 27 in a manner that is described below. Further, theportal server 20 has resident upon it or associated with it an API(Application Programming Interface) 21 that would allow externalapplications from external vendors, retailers and other agencies notpresent in any of the servers associated with system 10, to installtheir software/web applications. Validation procedures may be enforcedby the portal server to grant appropriate permissions to externalapplications to connect to system 10.

The users 12 of the system 10 may be any individual that has access to acomputing device 14. The computing device 14 is any computer typedevice, and may include a personal computer, laptop computer, handheldcomputer, phone, wearable computer, server type computer and any othersuch computing devices. The components of the computing device 14 in anexemplary embodiment are described in greater detail with regards toFIG. 2 to 56. The computing application 16 is a software applicationthat is resident upon or associated with the computing device 14. Thecomputing application 16 allows the user to access the system and tocommunicate with the respective servers. In an exemplary embodiment, thecomputing application aids in the rendering process that generates thethree-dimensional user model as is described below. In an exemplaryembodiment, the user accesses the system through a web browser, as thesystem is available on the Internet. Details on the web browser andcomputing application interaction are described with reference to FIG.10.

The communication network 18 is any network that provides forconnectivity between respective computing devices. The communicationnetwork 18 may include, but is not limited to, local area networks(LAN), wide area networks (WAN), an Intranet or the Internet. In anexemplary embodiment, the communication network 18 is the Internet. Thenetwork may include portions or elements of telephone lines, Ethernetconnections, ISDN lines, optical-data transport links, wireless datalinks, wireless cellular links and/or any suitable combination of thesame and/or similar elements.

The portal server 20 is a server-type computing device that hasassociated with it a server application 22. The server application 22 isa software application that is resident upon the portal server 20 andmanages the system 10 as described in detail below. The components ofthe software application 22 are described in further detail below withregard to FIG. 3. The retail server 24 is a server-type computing devicethat may be maintained by a retailer that has an online presence. Theretail server 24 in an exemplary embodiment has access to informationregarding various items of apparel that may be viewed upon thethree-dimensional model. The retail server 24 may be managed by anindependent third party that is independent of the system 10. Theretails server 24 may be managed by the portal server 20 and serverapplication 22. The community server 26 may be a server that implementscommunity networking sites with which the system 10 may interact. Suchsites may include sites where users interact with one another on asocial and community level. Through interacting with community server26, the system 10 allows for members of other online communities to beinvited to be users of the system 10. The entertainment server 23 in anexemplary embodiment, may be a server that provides gaming facilitiesand services; functions as a database of movies and music (new and oldreleases); contains movie related media (video, images, audio,simulations) and music videos; provides up-to-date information on movieshowtimes, ticket availability etc. on movies released in theatres aswell as on music videos, new audio/video releases; houses entertainmentrelated advertisement content etc. The media server agency 25 may belinked with media stations, networks as well as advertising agencies. Itincludes, but is not limited to news information, content and updates asrelates to events, weather, fashion, in an exemplary embodiment. Thefinancial institution server 27 in an exemplary embodiment may be linkedwith financial institutions and provides service offerings available atfinancial institutions and other financial management tools and servicesrelevant to online and electronic commerce transactions. These includefacilities for split-bill transactions, which will be described later.Services also include providing financial accounts and keeping track offinancial transactions, especially those related with the purchase ofproducts and services associated with system 10.

Reference is now made to FIG. 2, where a block diagram illustrating thecomponents of a computing device in an exemplary embodiment is shown.The computing device 14, in an exemplary embodiment, has associated withit a network interface 30, a memory store 32, a display 34, a centralprocessing unit 36, an input means 38, and one or more peripheraldevices 40.

The network interface 30 enables the respective device to communicatewith the communication network 18. The network interface 30 may be aconventional network card, such as an Ethernet card, wireless card, orany other means that allows for communication with the communicationnetwork 16. The memory store 32 is used to store executable programs andother information and may include storage means such as conventionaldisk drives, hard drives, CD ROMS, or any other non-volatile memorymeans. The display 34 allows the user to interact with the system 10with a monitor-type/projection-type/multi-touch display/tablet device.The CPU 36 is used to execute instructions and commands that are loadedfrom the memory store 32. The input devices 38 allow users to entercommands and information into the respective device 14. The inputdevices 38 may include, but are not limited to, any combinations ofkeyboards, a pointing device such as a mouse, or other devices such asmicrophones and multi-touch devices. The Peripheral devices 40 mayinclude, but are not limited to, devices such as printers, scanners, andcameras.

Reference is now made to FIG. 3, where a block diagram illustrating thecomponents of a server application is shown in an exemplary embodiment.The modules that are described herein are described for purposes ofexample as separate modules to illustrate functionalities that areprovided by the respective server application 22. The server application22 in an exemplary embodiment has associated with it a modeling module50, a community module 52, a management module 54, an environment module56, a retailer module 58, a shopping module 60, a wardrobe module 62 anadvertising module 64, entertainment module 66, and a financial servicesmodule 68. The server application 22 interacts with a data store 70 thatis described in further detail with regard to FIG. 4. The data store 70is resident upon the server in an exemplary embodiment and is used tostore data related to the system 10 as described below. Each of thesemodules may have a corresponding module on 14 and/or 16. Computationalload (and/or storage data) may be shared across these modules orexclusively handled by one. In an exemplary embodiment, the clothmodeling and rendering can be handled by the local application.

The modeling module 50, is used to generate a three-dimensional model ofa user. The user model as described below is generated based on a user'sphysical profile as provided through information of the user including,but not limited to images, movies, outlines of feet, hands, and otherbody parts; moulds/imprints including those of feet, hands, ears, andother body parts; scans such as laser scans; skin tone, race, gender,weight, hair type, high resolution scans and images of the eyes; motioncapture data, submitted measurements, and modifications made to thegenerated model. In an exemplary embodiment, the three-dimensional imagemay first be created based on one or more two-dimensional images thatare provided by the user (these include full body images and images ofthe head from one of more perspectives). These images are passed on to areconstruction engine to generate a preliminary three-dimensional model.In an exemplary embodiment, based on the respective images that areprovided, physical characteristics of the user are extracted. Thephysical characteristics are used to generate a preliminarythree-dimensional model of the user. This preliminary model is thenoptimized. In an exemplary embodiment of the optimization process, the3D surface of the preliminary model may be modified to better match theuser's physical surface. The modification to the mesh is made usingFinite Element Modeling (FEM) analysis by setting reasonable materialproperties (example stiffness) for different regions of the face surfaceand growing/shrinking regions based on extracted features of the face,Further, user-specified optimization is also performed. This process, inan exemplary embodiment, involves user specifications regarding thegenerated model, and further techniques described below. Users in anexemplary embodiment are asked for specific information relating totheir physical profile that is described in detail below. In exemplaryembodiment, the modeling module 50 combines the generatedthree-dimensional profile from the user's electronic image, with theuser-specified features and the user modifications to form athree-dimensional profile as is described in detail below. Users canupdate/re-build their model at a later point in time as well. This is toallow the user to create a model that reflects changes in their physiquesuch as growth, aging, weight loss/gain etc. with the passage of time.Additionally, the system 10 may be incorporated with predictionalgorithms that incorporate appropriate changes brought about by thegrowth and aging process in a given user model. Prediction algorithmsthat display changes in the user model after weight loss would also beaccommodated by system 10. These could be used by weight loss retailersto advertise their weight loss/health products. The user model can beincorporated with the personality or style aspects of the user or ofanother person that the user chooses. In an exemplary embodiment, usingcontent from a video that shows the user walking, system 10 can learnthe walking style of the user and apply it to the virtual model. Inanother exemplary embodiment, from an audio or video file of aconversation or a dialogue that a celebrity is engaged in, the accent ofthe celebrity may be learnt and applied to the speech/dialogues of themodel. In an exemplary embodiment, this can be accomplished usingbilinear models as discussed in paper 1 and 2.

The modeling module 50 also allows the user to view items of apparelthat have been displayed upon the user model that has been generated.The user is able to see how items of apparel appear on their respectivemodel, and how such items fit. The module enables photorealisticmodeling of apparel permitting life-like simulation (in terms oftexture, movement, color, shape, fit etc.) of the apparel. The modelingmodule 50 is able to determine where certain items of apparel may notfit appropriately, and where alterations may be required. Such adetermination is indicated to the user in exemplary embodiment throughvisual indicators such as, but not limited to, arrows on screen, varyingcolors, digital effects including transparency/x-ray vision effect wherethe apparel turns transparent and the user is able to examine fit in theparticular region.

The modeling module 50 also provides the user with the functionality totry on various items of apparel and for the simulated use of cosmeticproducts, dental products and various hair and optical accessories.Users are able to employ virtual make-up applicators to apply cosmeticproducts to user models. Virtual make up applicators act as virtualbrushes that simulate real cosmetic brushes can be used to selectproduct(s) from a catalogue (drag product) and apply (drop product) ontoa user model's face. This is accomplished, in exemplary embodiment, bywarping or overlaying the predefined texture map corresponding to theproduct on to the face using a technique similar to that used in [1].The texture map could be parameterized as a function of usercharacteristics such as skin tone, shape of face. The user is alsopresented with the option of letting the system apply selectedproduct(s) to the user model's face. In this case, the face texture mapis processed (using digital signal processing techniques as exemplaryembodiment) to create the effect of a given cosmetic product. Or, anadditional texture layer is applied with the desired effect on top ofthe existing face texture map. A correspondence between a cosmeticproduct and its effect on the user model allows users to visualize theeffect of applying a given cosmetic product (This also applies to hair,dental and optical products). Additionally, the module suggests the mostsuitable choice of cosmetic products as well as the procedure and toolsof application to enhance/flatter a user's look. Suggestions will alsobe provided along similar lines for dental, hair and optical products.Additionally, real-time assistance is provided to the user forapplication of cosmetic products. By connecting a webcam to system 10,the user can visualize themselves on their monitor or other displaydevice available while applying make-up (as in a mirror) and at the sametime interact with a real-time process that will be pre-programmed toact as a fashion consultant and will guide the user in achieving optimallooks and get feedback on their look as well while they apply make-up.In an exemplary embodiment, the application collects real-time video,image and other data from the webcam. Then, based on an assessment ofuser parameters such as face configuration, skin tone and type, facialfeature (eyes, nose, cheeks, chin etc.) configuration and type, theirrelative position and other parameters as well as based on theavailability of cosmetic products, the application provides text, audio,visual and/or other type of information to guide the user through theoptimal make-up application procedure given the specific parameters. Theuser can also specify other objective and subjective criteria regardingthe look they want to achieve such as the occasion for the look, thetype of look, the cosmetic product brands, time needed for applicationetc. The application provides specific feedback related to the existingmake-up that the user has already put on. For example, the applicationmay advise the user to use a matte foundation based on their skin type(program computes metrics involving illumination and reflectioncomponents based on the face image to assess the oiliness of the skin)or to use upward strokes while applying blush based on their cheekconfiguration (algorithms that localize contouring regions and/or assessconcavities on face regions are used). Additionally, the automaticmake-up applicator/advisor can present a virtual palette of cosmeticproducts on the monitor or display device and allow the users to selectthe colours/products of their choice. The program can perform a virtual‘make-over’ of the user. In an exemplary embodiment, the applicationuses the real-time video of the user available through the webcam orother forms of video/images captured by other forms of video/imagecapture devices; identifies the different facial features and appliesthe appropriate cosmetic products (cheeks with blush, eyelids with eyeshadow) to the video/image of the user and presents it on the display.If it involves streaming video content of the user, as in the case of awebcam, the user can visualize the cosmetic application process inreal-time as it is carried out by the application on the user's face onthe display. Instead of a pre-programmed application, a real fashionconsultant is also able to assist the user in a similar manner inachieving the desired looks with cosmetic products, using the webcamand/or other video or image capture feature. In an exemplary embodiment,the effect of applying cosmetic products can be achieved by moving theface texture map corresponding to the user model, or an image of theuser closer towards an average face. This can be accomplished byapplying PCA (Principal Components Analysis [2]) and removing the higherorder components, or it can also be done by computing the Fouriertransform of the user model's texture map or the user's image andremoving the higher frequency components. A similar technique can alsobe used to identify a user's beauty by looking at the weights of thehigher order principal components. Effect of applying beauty productscan be more realistically simulated by looking at the principalcomponents before and after the application of a cosmetic product on anumber of users and then applying the same change to the given user'stexture model or the user's image. The user can thus get assistance inapplying cosmetic products not simply on a 2D or 3D virtual effigy oftheir self but also on their actual face. This increases theinteractivity and precision of the cosmetic application process for theuser.

The user is also able to choose from various hairstyles that areavailable for selection. The modeling module 50 then causes the usermodel to be displayed with the hairstyle that has been selected by theuser. The user may change their hair style of the model, and apply hairproducts that affect the appearance of hair. The selections of hairstyles and other products by the user may be made based on hair stylesthat are featured from various respective hair salons. The moduleenables photorealistic modeling of hair permitting life-like simulation(in terms of texture, movement, color, shape etc.) of the model's hair.The modeling module 50 also allows the user to specify various actionsand activities that the user model is to undertake. The model may bemade to move in a variety of environments with various patterns ofmovement to provide to the user a better idea of how the model appearsin different settings or environments. The user is able to performvarious manipulations of the various parts of the user model in anexemplary embodiment. The user is presented in an exemplary embodimentwith specified activity choices that the user may wish the model toengage in. Examples of such activities include, but are not limited tosinging, speech and dancing. Where users wish to participate inactivities in shared environments where user models are allowed tointeract, the users in an exemplary embodiment join a network upon whichtheir models are placed into a common 3D environment. Any informationrelated to interaction between the user models such as location of themodel in the environment, occlusion, model apparel, motion/activityinformation related to the model is transmitted to each computingapplication either directly or via a server.

The community module 52 allows the user to interact with other users ofthe system 10 or with members of other community networks. The communitymodule 52 allows users to interact with other users through real-timecommunication. Messages can also be exchanged offline. The user caninteract with other users through their virtual character model. Themodel can be dressed up in apparel, make-up and hairstyles as desired bythe user and involved in interaction with other users. The user cananimate character expressions, movements and actions as it communicates.This is done via a set of commands (appearing in a menu or other displayoptions) to which the model has been pre-programmed to respond to. In anexemplary embodiment, a menu of mood emoticons (happy, angry, surprised,sad etc.) and action icons (wave, side-kick, laugh, salsa move, paceetc.) are presented to the user to enact on their virtual model whileusing it to communicate/interact with other users. Alternatively, theexpressions/movements/actions of the character model can be synchronizedwith the user's intentions which are communicated to the model in theform of text, speech, or other information. As an exemplary embodiment,the user may type or say the word laugh and the model will respond bylaughing. Another technique used for animating the model'sexpressions/movements/actions includes tracking the user'sexpressions/movements/actions through the use of a webcam, video camera,still camera and/or other video or image capture device and applying thesame expressions/movements/actions to the character model (synchronizedapplication or after a delay). The character may be programmed torespond to visual cues and/or expressions and/or tone and/or mood of theuser by putting on the appropriate expressions, acting accordingly anddelivering the effect of the user input. Further, speech or text inputto a user model may also be provided through a mobile phone.

The community interaction features of the system 10 allow the user toshare views of the user model with other users. By sharing the usermodel with other users, the user is able to request and receivecomments, ratings and general feedback regarding the respective apparelitems and style choices made by the user. Receiving feedback andcomments from other users enhances the user's experience with the systemby simulating a real world shopping experience.

When interacting with other users of the system 10, the community module52 allows users to interact with one another through use of theirrespective models. The community module 52 further includes chatfunctionality that allows users to participate in text, video or voicecommunication with other users of the system 10. (The chat applicationmay allow automatic translation to facilitate users who speak differentlanguages to communicate). Further, users may interact with other usersthrough engaging in collaborative virtual shopping trips as described indetail herein. Users can share their models with other users or buildmodels of other people and shop for items for other people too. Thisfeature would prove useful in the case of gift-giving. Another featurein this module includes a ‘hangout’ zone—a social networking, eventsplanning and information area. This is a feature which assists users inorganizing and coordinating social events, conferences, meetings, socialgatherings and other activities. Users can initiate new events oractivities in the hangout zone and send virtual invites to people intheir network and other users as well. The users can then accept ordecline invites and confirm if they can make it to the event.Event/activity/occasion information and description including, but notlimited to, details such as the theme, location, venue, participants,attendees, news and other articles related to the event, photos, videosand other event related media, user feedback and comments etc can beposted and viewed in the hangout zone. Suggestions on what to wearand/or bring to the event and where to buy it are also featured. Thiszone will also feature upcoming events and shows, music bands/groups andcelebrities coming to town. A map feature will be integrated to helpusers locate the venue of the event and get assistance with directions.The zone will also feature information on the area surrounding the venueof the event such as nearby restaurants, shopping plazas, other eventsin proximity of the venue etc. In another exemplary embodiment, groupsof users can coordinate excursion to movies. Users can start a newthread (i.e., create a new item page) in the hangout zone regardingvisiting the theatre on a particular date. Invitees can then vote forthe movie they want to watch, post news, ratings and other media itemsrelated to the movies; share views in celebrity or movie apparel on thepage; discuss and chat with other users regarding their plans.Information provided by the entertainment servers 23 and media agencyservers 25 will be used to keep content relating to movies, shows, andother entertainment venues updated in the hangout zone. In anotherexemplary embodiment, special events such as weddings and sports eventsmay be planned in the hangout zone, As an example, sample bridal outfitsmay be displayed in the zone for members of the group organizing thewedding, in the form of images, or on the virtual model of the bride oron mannequins etc. Apparel suggestions may be provided to the bride andgroom, for example, based on the season, time of day the wedding isheld, whether the event is indoor/outdoor, the budget allocated for theoutfits, etc. Suggestions on bridesmaids' dresses and other outfits maybe provided based on what the bride and groom are wearing and otherfactors such as the ones taken into account while suggesting bride andgroom outfits. A digital calendar may be featured in the hangout zoneindicating important timing information regarding the event such asnumber of days left for the event, other important days surrounding theevents etc. To-do and/or itemized lists which may be sorted according todays preceding the event may also be featured in the hangout zone. Afacility may be provided for incorporating information from othercalendars such as the Google™ Calendar™ or Microsoft™ Outlook™ etcand/or for linking these calendars within the hangout zone. A virtualassistant may be present in the hangout zone which is a 3D simulation ofa real or fictional character for purposes of providing information,help, and suggestions. The virtual assistant would be present to makeinteraction more ‘human’ in the hangout zone. In an exemplaryembodiment, an event profile page in the hangout zone is shown in FIG.43 displaying some of the features in the hangout zone. Animage/video/simulation 726 describing/related to the event can beuploaded on the page. The event title and brief information 727regarding the time, location, venue and other information related to theevent is displayed. A digital calendar is available to the moderators ofthe event for marking important dates and noting associated tasks. Anexample note 729 is shown that lists the important dates for the monthand which appears when the user clicks on the name of the month in thecalendar, in an exemplary embodiment, The note shows the number of daysleft for the event; the important dates and tasks associated with theevent as marked by the user. A facility is also available for members tojoin the event profile page to view the progress of preparation of theevent, take part in discussions and other activities surrounding theevent using the features and facilities available in the hangout zone.The member profile images/videos/simulations and/or name and/or otherinformation would be displayed in a panel 730 on the event page, in anexemplary embodiment. The viewer may scroll the panel using theleft/right control 731, shown in an exemplary embodiment to browse allmembers of the event. These members would also include the invitees forthe event. Invitations for the event can be sent to the invitees via thehangout zone. These members will be asked questions related to thestatus of their attendance such as if they plan to attend the event ornot, whether they are unsure or undecided and similar questions. Theresponses to these questions will be tallied and the total of eachresponse displayed as 732 in an exemplary embodiment. These responsescan also be used by the system to estimate costs incurred for the eventbased on attendance. Invitees may send the host or event planner (i.e.,the source of invitation) an RSVP confirming attendance via real-timenotification, email, SMS, phone, voice message, and similarcommunication means. The RSVP may contain other information such asaccompanying guests, outfit the invitee plans to wear, whether they needtransportation assistance in order to get to the event, tips for eventplanning and other such information related to the invitee with respectto the event. In the case of events where a registration fee isrequired, the system processes payments from the user. In cases wheredocuments are required for eligibility for attending the event (forinstance, a scientific conference), the system processes the documents.Upon selecting a member 733 from the event member panel 730, anotherwindow/dialog/pop-up 734 may appear with a larger image view of themember and details on member event status including fields such asattendance, member's event outfit, guest accompanying the invitee to theevent etc.; and/or member profile information. Icon 735 in thisdialog/pop-up window allows the member viewing the invitee's profile andevent status 734 to invite him/her on a shopping trip, via a real-timenotification, email, SMS, phone call or message and other means ofmessaging, while the icon 736 indicates if the invitee is online andallows the member viewing the invitee's profile to invite to chat orsend message to the invitee. Members on the event page can also getdetails of the venue and the area where the event is being held byclicking on the ‘area info’ section 737 as shown in an exemplaryembodiment. Upon doing so, a pop-up/dialog/window 738 opens up showinglocation and venue information on a map; places of interest in thevicinity of the event such as eateries, hangouts, and other scheduledpublic events. Further details on each of these different aspects may beobtained. A discussion forum facility 739 allows members of the event tostart topic threads and discuss various event related topics. Memberscan view all the discussion topics and categories, active members of thediscussion forum and view online members for engaging indiscussions/chats/real-time interaction with. Members in the hangoutzone can take advantage of the shopping and virtual modeling facilityavailable via system 10 to shop online for apparel and other needs forthe event. Invitees may shop for gifts via the electronic gift registryavailable as part of the event planning services. Shopping assistancepanels 741 and 742 provide tips, relevant event shopping and assistancecategories, display relevant advertisement and other information, andprovide other shopping help. Specific examples include event outfit, andgift ideas; listings, reviews and assistance in seeking event venue,organizers, decorators, fashion boutiques, car rentals etc. Reference isnow made to FIG. 44 which depicts some of the facilities in a browserwindow 745, that users can navigate to in the hangout zone, in anexemplary embodiment, The left and right panel menus, 746 and 747respectively, indicate some of the different online venues that the usercan visit on system 10. These include museums, studios, movies, parks,tours and other venues as well as stores, which will take the user tothe shopping module 60 on system 10. These facilities may be simulatedenvironments which users can visit or virtual events which users mayparticipate in via their virtual characters or directly. Alternatively,these facilities can be mapped to real physical venues which may beequipped with cameras and other visual equipment to facilitate real-timebrowsing and access to the facility via system 10. This would enablevirtual tourism and participation in real events in real-time fromremote locations either collaboratively with other users or on one'sown. In an exemplary embodiment, users may participate in a virtual tourof a real museum or a historical site. Users may watch a live video feed(or hear live audio feed) of a graduation ceremony or a musical concertor a hockey match or weddings and other community, social, business,entertainment, education events. Translation of video feeds in multiplelanguages is also available to members. Users can choose to view theevent in the original language or in the translated version.Translations may be provided by other members of the system in real-time(during live transmission) or after the event. Users can choose whichmember's translation to listen to during the event. Ratings of membertranslators may be available to guide this decision. Translations can beprovided either as subtitles or audio dubbing in an exemplaryembodiment. Translations may be computer-generated. This may be done inexemplary embodiment by converting speech to text, text to translatedtext, followed by translated text to speech in the new language.Furthermore, users can obtain information and details regarding specificreal events and/or places and/or facilities of interest to them such asmusic festivals, concerts, fairs and exhibitions, movie studios, games,historical sites etc in the hangout zone. For details on thesefacilities, refer to the environment module 56 and its descriptions inthis document. The facilities mentioned in FIG. 44 may manifestthemselves as the different types of environments described withreference to the environment module 56. A map facility 748 is availablewhich provides digital/animated representations of a virtual worldcontaining virtual facilities in the hangout zone and/or fictionalmappings of real facilities in virtual worlds. Real location and areamaps and venue information of the real places and events as well asdriving directions to events and venues are provided to assist users.The hangout zone may be linked to other websites that provide map,location and area information. Users can obtain assistance 749, whichmay be real-time/live, on what places they can visit, on what's new,special attractions, upcoming events, on activities in the hangout zoneetc. Users may send event invitations 750 to friends, as mentionedpreviously. These can be invitations for real events or events thatusers can participate in through system 10 such as games, virtual tours,virtual fashion shows and other events and activities. Users may examine751 other invitees to a particular event and see who else isparticipating in an event or activity or has confirmed attendance. Usersmay also obtain the latest weather and traffic updates 752 as well asall traffic and weather information relevant to a givenevent/venue/activity. Users may attend and participate in live virtualevents in real time where they can meet celebrities and get theirautographs signed digitally. The events described in the hangout zoneare not meant to be limited to the hangout zone or any specific spacebut are described as such in order to illustrate activities that can becarried out in a social networking space. The features described abovewith respect to the ‘hangout zone’ may be used as part of an eventmanagement module in the server application 22 whose services areavailable through a website or as part of a local application, Inaddition, the event management module may be used in conjunction orintegrated with a guest validation system. A guest validation systemwould assist in ascertaining if guests arriving at an event areconfirmed attendees or invitees to the event. Upon arriving at the eventvenue, guests can enter their name and password (which may be issuedwith the electronic invitation sent by the system, upon payment of eventregistration fees where required) either at a terminal or using theirhandheld. Alternatively, invitees can have a print out of an entry orinvitation card with a bar code (issued with the electronic invitation)which can be swiped at the event for entry. This would be most useful incases where an event requires registration and a fee to register.

This invention incorporates additional collaborative features such ascollaborative viewing of videos or photos or television and othersynchronized forms of multimedia sharing. Users may select and customizetheir viewing environments, and/or background themes and skins for theirviewer. They may select and invite other users to participate insynchronized sessions for sharing videos, and other multimedia. Inaddition to synchronized sharing, immersive features are provided bysystem 10 to further facilitate collaboration between users and to maketheir experience increasingly real and life-like as well as functionaland entertaining. During synchronized video sharing, for example, usersmay mark objects in the videos, write or scribble over the video contentas it plays, This feature can be likened to a TV screen that acts as atransparent whiteboard under which a video is playing and on top ofwhich markings can be made or writing is possible. During synchronizedmultimedia sharing, users can further interact by expressing emotionsthrough their character models which may be engaged in the sameenvironment or through emoticons and other animated objects. In anexemplary embodiment, if a funny scene is playing in a video, the usercan make their user model smile via a control key for their user modelwhich may be pre-programmed to respond with a smile when the givencontrol key is pressed. Pointing to objects, writing, expressingemotions through emoticons, SMS/text to invite for a shopping trip areactions as part of synchronized collaboration in an exemplaryembodiment. The whiteboard feature which permits freehand writing anddrawing may be available to users during shopping trips or events and/orfor any collaborative interaction and/or real time interaction and/orfor enabling users to take electronic notes and/or draft shopping listsand uses described with reference to FIG. 20 in this document. Based onthe content of the whiteboard deciphered through OCR optical characterrecognition techniques or sketch to model recognition [3] or speech tomodel recognition, related content (for example advertisements) may beplaced in the proximity of the drawing screen and/or related content maybe communicated via audio/speech, and/or graphics/images/videos.

A ‘virtual showcase’ will allow users to showcase and share their talentand/or hand-made items (handiwork) and/or hobbies with online users. Inan exemplary embodiment, users can upload digital versions of their artwork which may include any form of art work such as paintings orhandicrafts such as knit and embroidered pieces of work; handmadeproducts such as wood-work, origami, floral arrangements; culinarycreations and associated recipes; and any form of outcome or product orresult of a hobby or sport. All the above are meant to be exemplaryembodiments of items that can be displayed in the virtual showcase. Asfurther exemplary embodiments, users can post/showcase videosdemonstrating feats of skateboarding or instructional videos oranimations for cooking, and other talents. The virtual showcase maycontain virtual art galleries, in an exemplary embodiment, featuringart-work of users. Members may be able to browse the virtual art galleryand the gallery environment may be simulated such that it gives theusers the illusion of walking in a real art gallery. The art galleriesmay be simulated 2D or 3D environments, videos, images or anycombination thereof and/or may include components of augmented reality.Users can also adorn their virtual rooms and other 2D or 3D spaces withtheir virtual artwork.

The management module 54 allows the user to control and manage theiraccount and settings associated with their account. The user may resethis/her password and enter and edit other profile and preferenceinformation that is associated with the user. The profile and preferenceinformation that is provided by the user may be used to tailor apparelitems, or combinations of apparel items for the user.

The environment module 56 allows the user to choose the virtualenvironment in which to place their user model. As the system 10 allowsusers to visualize how various apparel items will appear when they arewearing them, the ability to choose respective virtual environmentsfurther aids the user in this visualization process. For example, wherea user's 3-D model is used to determine the suitability of evening wearor formal wear, the user is better able to appreciate the modeling wherea formal background is provided. The virtual environments may be staticimage or dynamic backgrounds or three-dimensional or multi-dimensionalenvironments, or any suitable combination of the above. In an exemplaryembodiment, a dynamic background could include an animated sequence or avideo or a virtual reality experience. Images or animations or video orother multimedia that are represented by the respective environments mayinclude, but are not limited to, vacation destinations, touristdestinations, historical sites, natural scenery, period themes (the 60s,70s, Victorian era etc.), entertainment venues, athletic facilities,runways for modeling, etc. The environments that are provided by thesystem 10 may be customized and tailored by the users. Specifically,users may be provided the option of removing or adding componentsassociated with the environment and to alter backgrounds in theenvironments. For example, with respect to adding and or removingphysical components, where a living room environment is being used andis provided to the system 10, various components associated with theliving room may be added, deleted or modified. With respect to theaddition of components, components such as furniture and fixtures may beadded through functionality provided to the user. The user in anexemplary embodiment is provided with drag and drop functionality thatallows the user to drag the various components into an environment, andout of an environment. The drag-and-drop functionality may incorporatephysics based animation to enhance realism. Optionally, the users mayspecify where things are placed in an environment. In an exemplaryembodiment, the users are able to choose from a listing of componentsthat they wish to add. As described below, the respective componentsthat are chosen and placed in the virtual environments may be associatedwith respective companies that are attempting to promote their products.For example, where a user has placed a sofa in their virtualenvironment, the user may view the selections of sofas that may beplaced in the virtual environment and each sofa that may be selectedwill have information pertaining to it that will help the user decidewhether to place it in their virtual environment. Through partneringwith the system 10, retailers of non apparel items can increase exposureto their product offerings. Advertisements may be displayed in theseenvironments and thus, these environments would serve as an advertisingmedium. For example, a billboard in the background may exhibit a productad or people in the environment may wear apparel displaying logos of thebrand being advertised. There could also be theme-based environments toreflect the nature of the advertising campaign. For example, a companyselling a television with a new-age look may find the use of anenvironment with a futuristic theme useful for advertising.

Virtual environments may also represent or incorporate part or whole ofa music video or movie or game scene or animation or video. User modelswould have the ability to interact with virtual embodiments of moviecharacters and celebrities. As an example, the user model may be placedin a fight scene from a movie. Another feature that would be supportedby the entertainment environments is to allow users to purchase appareland other items shown in the particular movie. For example, the usercould purchase apparel worn by the characters in the movie or the carsdriven in the movie or the mobile phones used in the movie.Additionally, users could replace the characters in the movie or musicvideo with their user models. The model would be able to orchestrate theexact movements (dialogue, movements, actions, expressions) of theoriginal character. This would involve facial animation and lip syncingof the user model to replicate expressions and facial movements of theoriginal character. Furthermore, the movements of the original charactercan be extracted, in exemplary embodiment, either manually or usingmachine learning algorithms (for example: pose tracking and pose recovertechniques) and then applied to the user model. For purposes ofincreasing computational efficiency, the system 10 may provide the userwith pre-rendered scenes/environments where the music and environmentcannot be manipulated to a great degree by the user but where renderingof the character model can occur so that it can be inserted into thescene, its expressions/actions can be manipulated and it can be viewedfrom different camera angles/viewpoints within the environment. Userscan save or share with other users the various manifestations of theiruser model after manipulating/modifying it and the animation/videosequence containing the model in various file formats. The modified usermodel or the animation/video sequence can then be exported to otherlocations including content sharing sites or displayed on the profilepage. In an exemplary embodiment, the user may indicate their displaystatus through the use of their character model with the appropriatebackdrop and other digital components. For instance, users may indicatethat they are reading a given book by displaying their model on theirprofile page reading a book against a backdrop that reflects the themeof the book or their model may be engaged with other models in an actfrom the book or a play or a movie that they are watching. Anotherfeature that the virtual environments along with the user models affordto the user is the ability to take studio portraits of their respectiveuser models with the different environments serving as backdrops. Userscan also invite friends and family for group portraits with theirmodels. Features will also be present to add effects and/or enhance theportrait photos or apply various artistic styles (for example, antiquelook, watercolour effect etc.) and perform various othernon-photorealistic renderings.

A feature encompassing a virtual space/environment where virtual fashionshows are held is available through system 10. Professional and amateurdesigners can display their collections on virtual models in virtualfashion shows. The virtual models and virtual environments can be custommade to suit the designer's needs and/or virtual models of real usersand celebrities may be employed. Auctions and bidding can take place inthese virtual spaces for apparel modeled in the fashion shows. Groups ofusers can also participate in virtual fashion shows in a sharedenvironment using their 3D models to showcase apparel.

The whole or part of a virtual environment may incorporate physics basedanimation effects to enhance realism of the environment, its contentsand interaction with the user. In an exemplary embodiment, anenvironment representing a basketball court could be integrated withphysics based animation effects. In this case, the motion dynamics ofthe basketball players, the ball, the basket etc. would be based on thephysics of real motion and thus, the game sequence would appearrealistic. Users are also able to select their own environment, and mayupload their own environment to be used in the system 10. Furthermore,the system 10 also includes simulated shopping environments. An animatednavigation menu is provided so that the user may locate stores/stalls ofinterest. The shopping environment, in an exemplary embodiment, may berepresented by components of a virtual mall which may containsimulations of components of real stores, or it may be a simulatedrepresentation of a real mall which may contain other animated virtualcomponents. As the user browses the shopping environment, theenvironment may be presented as a virtual reality animation/simulationwhich may contain video/simulations/images of actual/real stores andcomponents; or it may be presented as a real-time or streaming video ora video/series of images of a real mall with animated stores andcomponents; or as a virtual reality simulation of a real store. System10 recommends stores to visit based on specific user information such asprofession, gender, size, likes/dislikes etc. For instance, for a shortfemale, the system can recommend browsing petite fashion stores. Basedon a user's apparel size, the system can point out to the user if aproduct is available in the user's size as the user is browsing productsor selecting products to view. The system may also point out theappropriate size of the user in a different sizing scheme, for example,in the sizing scheme of a different country (US, EUR, UK etc.). Insuggesting appropriate sizes to user in products that may vary accordingto brand, country, and other criteria, the system also takes intoaccount user fit preferences. For instance, a user may want clothes tobe a few inches looser than his/her actual fit size. In an exemplaryembodiment, the system would add the leeway margin, as specified by theuser, to the user's exact fit apparel size in order to find the desiredfit for the user. As described below, a user who wishes to view and/ormodel apparel items may select from the various items of apparel througha shopping environment such as a store or a mall. In these respectiveenvironments, the models are allowed to browse the virtual storeenvironment by selecting and inspecting items that are taken from therespective racks and shelves associated with the virtual environment. Inthe shopping environment, physics based animation can be incorporated tomake the shopping environment, its contents and user interaction withthe environment realistic. In an exemplary embodiment, the clothes inthe shelves and racks can be made to appear realistic by simulating realtexture and movement of cloth. Additionally, a live feed can be providedto users from real stores regarding the quantity of a particular item.This information can either be conveyed, for example, either numericallyor an animation of a shelf/rack containing the actual number of items ininventory can be displayed or a video of the real store with the itemson shelf can be displayed to the user. The live feed feature can be usedby the source supplying the apparel to convey other information such asstore/brand promotions, special offers, sales, featured items etc. (notrestricted to real-time inventory information). Furthermore, theshopping environment can include other stores and fixtures and otheritems found in a real shopping mall to simulate/replicate real shoppingenvironments as closely as possible. In an exemplary embodiment, foodstores and stalls may be augmented in the virtual shopping environment.These ‘virtual food stores’ could represent simulations or images/videosof fictional or non-fictional stores. These virtual stores would serveas an advertising medium for food brands and products as well assuperstores, restaurants, corner stores or any other place providing afood service, manufacturing or serving as the retail outlet for a foodbrand. There could be virtual ads, products and promotions being housedin these virtual stores. Additionally, these could be linked to actualproduct and store sites. Virtual characters acting as store personneloffer virtual samples of ‘featured food products’, just as in a realmall setting. Other items found in real shopping environments that areincorporated include fountains, in an exemplary embodiment. Thesevirtual fountains can be incorporated with physics based animationtechniques to simulate water movement as in a real fountain. Storepersonnel such as sales representatives and customer servicerepresentatives are represented by virtual characters that provideonline assistance to the user while shopping, speak and orchestratemovements in a manner similar to real store personnel and interact withthe user model. An ‘augmented reality display table’ is featured bysystem 10 where vendors can display their products to the customer andinteract with the customer. For example, a jewelry store personnel maypick out a ring from the glass display for showing the user. Asalesperson in a mobile phone store may pick out a given phone anddemonstrate specific features. At the same time, specifications relatedto the object may be displayed and compared with other products. Usersalso have the ability to interact with the object in 2D, 3D or higherdimensions. The salesperson and customer may interact simultaneouslywith the object. Physics based modeling may also be supported. Thisdisplay table may be mapped to a real store and the objects virtuallyoverlaid. In some real malls, one can also find indoor game facilitiessuch as ice-skating rinks, golf parks, basketball etc. Environments thatsimulate these facilities virtually will be available. Users can engagetheir models in these activities and participate in a game with othersusers. As in a real mall, the user can see other ‘people’ in a virtualmall. These may represent real users or fictional virtual characters.The user will have the option to set their user model as invisible orvisible so that their model can be viewed by other users browsing themall.

In an exemplary embodiment, this collaborative environment works asfollows: The local application 271 provides a visualization engine.Webcam content from the customers and the sales personnel may beintegrated into or used in conjunction with the engine. If 3D productmodels are available, they can be used interactively via the common modeor other modes of operation, as discussed with reference to FIG. 7, forexample. If product models are unavailable, then webcam views may beused either directly or converted to models based on webcam images(using techniques similar to those discussed in [3] for going fromsketch to model in exemplary embodiment). These models/images can thenbe used in the visualization engine. Interaction with the engine cantake place using conventional input/output (I/O) devices such as akeyboard and a mouse, or using I/O devices discussed with reference toFIG. 54. Video capturing devices may be used to capture the view of acounter or a product display in the store, for example. This content maybe transmitted both to the salesperson and the customer. Either partycan then augment this content with their own input. The customer mayalso bring in objects into this augmented world, for example, for colouror style matching. Augmentation may be accomplished using techniquessimilar to those in [4]. The collaborative environment described herewith reference to FIG. 36 may be thought of as a 3D version of thecollaborative environment described with reference to FIG. 20. All ofthe tools available in the collaborative environment discussed withreference to FIG. 20 may be available in the collaborative environmentof FIG. 36.

The various respective virtual environments that are used, may all haveassociated with them various multimedia files that may be linked to therespective environments. For example, music, or video files may belinked or embedded into the respective environments. Also, the system 10may also allow for downloading of music (and other audio files) from arepository of music, in an exemplary embodiment, that may then be playedwhile the user is navigating and/or interacting with their respectiveenvironment. The user will have the option of selecting music from therepository and downloading tracks or directly playing the music from amedia player within the browser. Additionally, audio files can also runseamlessly in the environment. These can be set by the sponsor of anenvironment. For example, in a virtual music store environment, thestore sponsor can play tracks of new releases or specials beingadvertised. In another exemplary embodiment, in a movie sceneenvironment, the soundtrack of the movie could play within theenvironment. These tracks (playlist content, order of tracks, lengthetc.) can be customized according to the sponsor or user. The sponsor ofthe environment and the music or media files sponsor do not necessarilyhave to be the same. Additionally, the user may be given control overthe type of media files that are played within or linked with anenvironment. Instead of a repository of audio files, the medium may alsobe an online radio, The radio may be mapped to real radio stations.Users have the option to share media files (name, description and otherinformation associated with the file and/or actual content) with theirsocial network or send links of the source of the media files. Users canalso order and purchase media files that they are listening to online.In an exemplary embodiment, a ‘buy now’ link would be associated withthe media file that would take the user to the transaction processingpage to process the purchase of the media file online.

Users may create their own 3D or 2D virtual spaces by adding virtualcomponents from catalogues. In an exemplary embodiment, a user may rentor buy virtual rooms (2D or 3D) from a catalogue and add virtualfurniture, virtual artwork, virtual home electronics such as a TV,refrigerator, oven, washing machine, home entertainment system etc. andother components. The user may add rooms to create a home with outdoorextensions such as a patio and backyard to which components may also beadded. Users may visit each other users' virtual spaces andenvironments. Users may also buy virtual food products. which may bestored in virtual refrigerators or stores. These virtual food productsmay be designed such that they decrease over time and eventually finishor become spoilt if unused ‘virtually’. This would help kids orteenagers, for example, to understand the value of food, its lifecycle,handling and storage and other facts. Furthermore, the proceeds from thepurchase of virtual food could be used to sponsor aid in developingcountries. In an exemplary embodiment, purchasing a bag of virtual ricemay be equivalent to donating a bag of virtual rice as food aid todeveloping countries. Users may furnish their rooms with objects thatchange or grow with time such as plants. The user may buy a virtual seedand over time, the seed would grow into a full-size virtual plant. Thevirtual plant may be designed such that it grows automatically or uponproper caretaking by the user such as providing virtual water,nutrients, sunlight and other necessities to the plant. This would helpusers to become more empathic and acquire useful skills such asgardening or caretaking. Florists and greenhouses may also find thisfeature useful. They may design virtual plants and flowers such thattheir requirements are mapped to the real plants or flowers theyrepresent. For instance, roses may require specific nutrients, soiltypes, sunlight duration etc. for their proper growth. In an exemplaryembodiment, virtual rose plants may be designed to grow only if providedwith the necessities (virtual) that real roses require. Thus, thesevirtual plants would prove useful as instructional or training tools forpeople who would like to learn how to cultivate specific plants properlybefore purchasing real plants. Depending on how they raise their virtualplants, users may be given scores. Users would also be able to purchasethe real plants from florists, greenhouses and other stores subscribingto system 10, whose information would be available to users.Furthermore, users may buy virtual pets. These virtual pets may bedesigned to grow on their own or upon proper caretaking by their ownersjust as in the case of virtual plants. This feature could help users tobecome better pet caretakers before they buy real pets. The concept ofvirtual pets can be taken further. Proceeds that are collected from thepurchase of virtual pets may be used to support animal shelters orhumane societies or animal relief or wildlife conservation efforts. Avirtual pet may be mapped to an animal that has been saved as a resultof the proceeds collected from the purchase of virtual pets. Users maydirectly sponsor an animal whose virtual representation they would ownupon sponsoring the animal. Users would also receive updates about thewelfare of the animal they sponsored (if they are not able to directlyown the real animal such as in the case of a wild animal) and aboutrelated relief, rescue or conservation efforts associated with similaranimals.

The retailer module 58 allows the system 10 to interact with the variousrespective retailers with which the system 10 is associated.Specifically, the retailer module 58 tracks the respective items thatmay be purchased through use of the system 10. The retailer module 58interacts with the retail servers 26 of retailers with respect toproduct offerings that may be available through the system 10.Information from the retailer module 58 pertaining to items that can bepurchased is acquired by system 10. This information may be encapsulatedin a CAD (Computer Aided Design) file for example.

The shopping module 60 allows for users to purchase items that may beviewed and/or modeled. Each retailer in the retailer module 58 may havea customizable store page or virtual store available in the shoppingmodule 60. Users can administer their page or virtual/online store asdiscussed with reference to FIG. 42. Each store can be customizedaccording to the retailer's needs. Retailers may add web and softwarecomponents to their store available through system 10. These componentsinclude those that would allow the retailer to add featured items,special offers, top picks, holiday deals and other categories of itemsto their virtual store. The retailer can make available their productsfor sale through these stores/pages. The users of the system 10 asmentioned above have access to various online product catalogues fromvirtual stores and/or virtual malls. These catalogues may be mapped fromvirtual stores and/or virtual malls or real stores and/or malls. Theuser will be asked specific information relating to the shoppinginterests and style preferences. The shopping module 60, based on theuser-specified preferences and information may also make recommendationsregarding items of apparel that are based on the user's interests,preference and style that have been determined from previous purchases.This can be accomplished using a variety of machine learning algorithmssuch as neural networks or support vector machines. Currentimplementation includes the use of collaborative filtering [5].Alternatively, Gaussian process methodologies [6] may also be used. Inan exemplary embodiment, using Gaussian process classification,recommendations are made to the user based on information collected onthe variables in the user's profile (example: preferences, style,interests) as well as based on the user's purchasing and browsinghistory. Moreover, the uncertainty that is computed in closed form usingGaussian process classification is used to express the degree ofconfidence in the recommendation that is made. This can be expressedusing statements like ‘you may like this’ or ‘you will definitely lovethis’ etc. The interests of the user may be specified by the user, andalternatively may be profiled by the system 10 based on the user'sdemographics. The shopping module 60 also provides the user with varioussearch functionalities. The user may perform a search to retrieveapparel items based on criteria that may include, but are not limitedto, a description of the apparel including size, price, brand, season,style, occasion, discounts, and retailer. Users can search and shop forapparel based on the look they want to achieve. For example, this couldinclude ‘sporty’, ‘professional’, ‘celebrity’ and other types of looks.Users may also search and shop for apparel belonging to specialcategories including, but not limited to, maternity wear, uniforms,laboratory apparel etc. Apparel may be presented to the user on virtualmannequins by the shopping module 60. Other forms of display include a‘revolving virtual display’ or a ‘conveyor belt display’ etc. In anexemplary embodiment, a revolving display may assume the form of aglass-like cube or some other shape with a mannequin on each face of thecube/shape showcasing different apparel and/or jewelry. In anotherexemplary embodiment, a conveyor belt display may feature virtualmannequins in a window, donning different apparel and/or jewelry. Themannequins may move in the window in a conveyor belt fashion, with asequence of mannequin displays appearing in the window periodically. Thespeed of the conveyor belt or the revolving display may be modified.Other displays may be used and other manifestations of the conveyor andrevolving display may be used. For instance, the mannequins may bereplaced by user models or by simply product images and/or othervisual/virtual manifestations of the product. Reference is now made toFIG. 45 where another display scheme—the ‘Style browser’ 755 is shown inan exemplary embodiment, The style browser display operates directly onthe user model 650 in that the apparel items in an electronic catalogueare displayed on the user model as the user browses the productcatalogue. For example, in the display window 755, the user can browsetops in a catalogue in the window section 756 by using the left 757 andright 758 arrow icons. As the user browses the catalogue, the tops aremodeled and displayed directly on the user model 650. Thus, the user isable to examine fit and look information while browsing the catalogueitself. In a similar fashion, the user can browse skirts and pants inthe display section 759; shoes in section 760; accessories likeearrings, cosmetics and hairstyles in section 760. Right-clicking on agiven display section would make available to the user the categories ofapparel that the user can browse in that section, in an exemplaryembodiment. Displayed apparel (whether in shopping environments, storesor electronic catalogues) may be in 2D or 3D format. Users can also viewdetailed information regarding apparel. For example, this informationincludes material properties of the apparel such as composition,texture, etc; cloth care instructions; source information (country,manufacturer/retailer); images describing apparel such as micro-levelimages that reveal texture; etc. Other information assisting the user inmaking purchasing decisions may also be displayed. For example, user andcustomer reviews, ratings,manufacturer's/retailer's/designer's/stylist's notes etc. The displayinformation for each apparel will also include the return policy forthat item. This policy may include terms that are different in the casethat an item is returned via postal mail versus if the item is taken toa physical store location for return by the customer. In an exemplaryembodiment, for the latter case, the return policy may be mapped to theterms and conditions of the physical store itself. This would allow auser to purchase something online and still be able to return it at aphysical store location. Alternatively, the retailer may specify adifferent return policy for the apparel when it is bought online asopposed to when it is bought at the physical store. The return policymay also incorporate separate terms and conditions that take intoaccount the requirements of system 10 for returning any given item. Asusers are shopping, matching/coordinating items that go with the itemsthe users are looking at or items that are in the users fitting room,shopping cart, or wardrobe, and that fit the users body and their taste,may be presented to the users. Suggestions on coordinating/matchingitems may also be made across users. For example, if a bride and abridegroom go on a shopping trip, a wedding dress for the bride and acorresponding/matching tuxedo for the bridegroom that fit themrespectively may be presented.

At any time while browsing or viewing products, the user may choose totry on apparel of interest on their user model to test the fit ofapparel. In order to facilitate this process, a virtual fitting room isavailable to the user. The virtual fitting room includes items that theuser has selected to try on or fit on their user model and that the usermay or may not decide to purchase. In exemplary embodiment, the fittingroom provides the user with a graphical, simulated representation of afitting room environment and the apparel items selected for fitting onthe user's model. The user can add an item to their fitting room byclicking on an icon next to the item they wish to virtually try on. Oncean item has been added to the fitting room, that item will becomeavailable to the user in the local application for fitting on theirmodel. An example of user interaction with the fitting room isillustrated in FIG. 27. While browsing apparel catalogues or viewingsuggested apparel items by system 10, the user may choose to add an itemto the fitting room for trial fit with their user model. Once the itemhas been added to the fitting room, the user may try on the item ontheir user model, and/or decide to purchase the item, in which case theapparel item can be added to the virtual wardrobe described later.Alternately, the user may decide not to purchase the item in which casethe item will stay in the fitting room until the user chooses to deleteit from their fitting room. Users may make the contents of their fittingroom publicly accessible or restrict access to members of their socialnetwork or provide limited access to anyone they choose. This optionwill allow users to identify items of interest that other users have intheir fitting room and browse and shop for the same or similar items onsystem 10. Physics based animation can be incorporated to make thefitting room, its contents and user interaction with the fitting room asrealistic as possible. In exemplary embodiment, the clothes in thefitting room can be made to appear realistic by simulating real textureand movement of cloth. With regards to interaction with the digitalapparel, accessories and other components, users may be able to drag anddrop clothes, optical accessories, hairstyles, other apparel,accessories, and digitized components and their manifestations ontotheir character model. In one exemplary embodiment, they will be able todrag components placed in the fitting room or wardrobe or from anelectronic catalogue onto their model. The drag-and-drop functionalitymay incorporate physics based animation to enhance realism. Optionally,the users may specify where things are placed on their character model.At any time while browsing or viewing products or trying apparel ontheir user model, the user may choose to order and purchase the realapparel online. The user may also submit fit information (visual as wellas text) including information on where alterations may be needed, asprovided by the modeling module 50, as well as any additionalinformation associated with an apparel item that the user is purchasingonline to a ‘tailoring’ service. This service would be able to make therequisite alterations for the user for a fee. A facility would also beavailable to the user to custom order clothes online from a designer orsupplier of apparel if they (designer, supplier) choose to provide theservice. In the case of purchasing gifts for other people, the user maybuild a model for the person for whom the gift is intended and fitapparel on to this third party model to test goodness of fit beforepurchasing the apparel. If the user for whom the gift is being purchasedalready has a user account/profile available in system 10, then theiruser model may be accessed by the gift-giver upon receiving permissionfrom the user for purposes of testing goodness of fit. If a user wishesto access fit or other information or the user model of a friend, thefriend would receive a notification that the specific information hasbeen requested by the user. The friend would have the option to grant ordeny access to any or all of their information or their user model. Ifthe friend denies access, the user may still be able to purchase a giftfor the friend as the system will be able to access the friend'sinformation and inform the user if a particular apparel is available intheir friend's size. The system would, thus, provide subjectiveinformation regarding the fit of an apparel with respect to another userwithout directly revealing any fit or other information of the user forwhom the item is being purchased. If an apparel item is available in thefriend's size, the user may order it upon which the system would deliverthe appropriate sized apparel (based on the sizing and fit informationin the friend's profile) to the friend. A confirmation request may besent to the friend for confirming the size of the apparel before thepurchase order is finalized. (This method can be used for other productssuch as prescription eyewear). Users have the option to display icons ontheir profile and/or home page that indicate gifts received from otherpeople (items purchased on the site for respective user by other users).A ‘Mix and Match’ section will allow users to view items from differentvendors. This could be, for instance, for purposes of coordinatingdifferent pieces of apparel (for example tops, bottoms, jewelry, bags).Users may coordinate items and visualize their appearance on the usermodel. This visualization would assist users in the mix and matchprocess. Items on sale may also be presented from different vendors inthe mix and match section. Items on sale/discounted items may also bepresented in other areas of the site. Furthermore, there may be othersections on the site featuring special items available for purchase. Inexemplary embodiment, these may include autographed apparel and othergoods by celebrities. Not only is the user able to purchase real apparelfrom the site (described later on), but the user can also buy virtualmanifestations of apparel, hairstyles, makeup etc. Users may beinterested in purchasing these virtual items for use in external sites,gaming environments, for use with virtual characters in otherenvironments etc. Users can also search for and buy items on otherusers' shopping lists, registries and/or wishlists. Users may alsoset-up gift registries accessible on their member pages for occasionssuch as weddings, anniversaries, birthdays etc.

The shopping module 60 also determines for each user a preferred orfeatured style that would be suitable for the respective user. Thedetermination of a preferred or featured style may be based on variousinputs. Inputs may include the preferences and picks of a fashionconsultant of which the system 10 keeps track. The one or more fashionconsultant's choices for featured styles may be updated into the system10, and the system 10 then provides respective users with updated stylechoices based on the selections of the fashion consultants. Also, stylesand/or apparel items may be presented to the user based on informationthe system 10 has collected regarding their shopping preferences,stores, brands, styles and types of apparel that are purchased, alongwith personal information related to their physical profile and age. Inaddition, the user model may be used to make apparel suggestions by thesystem. In an exemplary embodiment, the convex hull of the user model isused to determine apparel that would best fit/suit the user. The variousfeatured looks that are selected by the system 10 may be presented tothe user upon request of the user, and the selected featured looks mayalso be presented to the user upon login to the system. Also, variousselected styles with a user's model may be presented to the user uponrequest or upon login where the user model is modeling apparel that issimilar to what celebrities or other notable personalities may bewearing. Fashion consultants, stylists and designers may be available onsite for providing users with fashion tips, news, recommendations andother fashion related advice. Live assistance may be provided through achat feature, video and other means. Additionally, it may be possiblefor users to book appointments with fashion consultants of their choice.Animated virtual characters representing fashion consultants, stylistsand designers may also be used for the purpose of providing fashionrelated advice, tips news and recommendations. Virtual fashionconsultants may make suggestions based on the user's wardrobe andfitting room contents. It would also be possible for users interested ingiving fashion advice to other users to do so on the site. In anexemplary embodiment, this may be accomplished by joining a ‘fashionamateurs’ network where members may provide fashion advice to otherusers or even display their own fashion apparel designs. Consultants maybe available to provide assistance with other services such astechnical, legal, financial etc.

The wardrobe module 62 provides the user with a graphical, simulatedrepresentation of the contents of their real and/or virtual wardrobe.The virtual wardrobe comprises the respective items of apparel that areassociated with the user in the system 10. For example, the virtualwardrobe will store all of the items that the user has purchased. FIG.27 describes an instance of user interaction with the virtual wardrobe440 and fitting room 420. The user may browse apparel 400 displayed bythe system, an instance of which is described with reference to FIG. 22.Once the user decides to purchase an item, it will be added to thevirtual wardrobe. The user may then choose to keep the item in theirwardrobe or delete it. If the user decides to return an item, that itemwill be transferred from the user's wardrobe to the fitting room. Thevirtual wardrobe may also comprise representations of apparel items thatthe user owns that are not associated with the system 10. For example,the user may upload respective images, animation, video and othermultimedia formats or any combination thereof of various real apparelitems to the system 10. Once uploaded, the users are then able tointeract with their respective physical wardrobe contents through use ofthe system 10. Identification (ID) tags on the virtual wardrobe itemsmay assist the user in mapping items from the real to virtual wardrobe.An ID tag can have standard or user defined fields in order to identifya given item. Standard fields, for instance, can include, but are notlimited to, ID number, colour, apparel type, occasion, careinstructions, price, make and manufacturer, store item was purchasedfrom, return policy etc. User defined fields may include, for example,comments such as ‘Item was gifted to me by this person on this date’,and other fields. Users are able to browse the contents of theirwardrobe online. This allows the user the ability to determine whichapparel items they may need to purchase based on their need and/ordesire. Users may make the contents of their wardrobe publiclyaccessible or restrict access to members of their social network orprovide limited access to anyone they choose. This option will allowusers to identify items of interest that other users have in theirwardrobe and browse and shop for the same and/or similar items on thesystem 10. An icon may appear on the profile/home page of the user—‘buywhat this user has bought’ to view recent purchases of the user and buythe same and/or similar items via system 10. The user may also decide toconduct an auction of some or all of the real items in their wardrobe.In such a case, the user will be able to mark or tag the virtualrepresentations of these items in their virtual wardrobe and other userswith access to the wardrobe can view and purchase auction items ofinterest to them. In exemplary embodiment, an icon may appear on theprofile page of the user indicating that they are conducting an auctionto notify other users. It may be possible for users to mark items intheir virtual wardrobe for dry-cleaning. This information may be used tonotify dry-cleaning services in the area about items for pick-up anddelivery from respective users in an exemplary embodiment. Physics basedanimation can be incorporated to make the wardrobe, its contents anduser interaction with the wardrobe as realistic as possible. Inexemplary embodiment, the clothes in the wardrobe can be made to appearrealistic by simulating real texture and movement of cloth.

Users may organize their virtual wardrobe contents according to variouscriteria. The wardrobe classification criteria may include, but are notlimited to, colour, style, occasion, designer, season, size/fit,clothing type, fabric type, date of purchase etc. By indexing theapparel items that belong to the user according to various criteria, theuser may then be able to determine through various search criteria whatitems of apparel to wear. The virtual wardrobe may also have associatedwith it multimedia files such as music, which provide a more enjoyableexperience when perusing the contents of the virtual wardrobe. Avirtual/real style consultant and/or other users may be available toadvise on the contents of the wardrobe.

The advertising module 64 in an exemplary embodiment coordinates thedisplay and use of various apparel items and non-apparel items.Advertisers associated with the system 10 wish for their particularproduct offering to be displayed to the user in an attempt to increasethe product's exposure. The advertising module determines which offeringassociated with an advertiser is to be displayed to the user. Somecomponents related to the advertising module 64 are linked to theenvironment module, the details of which were discussed in the sectiondescribing the environment module 56. These include, in exemplaryembodiments, environments based on a theme reflecting the product beingadvertised; components associated with environments such asadvertisement banners and logos; actual products being advertisedfurnishing/occupying the environments. Music advertisers can linkenvironments with their playlists/soundtracks/radio players. Movieadvertisers can supply theme based environments which may featuremusic/apparel/effigies and other products related to the movie. Userswill be able to display character models on their profile page wearingsponsored apparel (digitized versions) that sponsors can make availableto users through the advertising module 64; or users can display imagesor videos of themselves in their profile wearing real sponsored apparel.In a similar manner, users supporting a cause may buy real or digitalapparel sponsoring the cause (for example, a political or charitablecause) and display their character model in such apparel or put upvideos or images of themselves in real versions of the apparel.Advertisers belonging to the tourism industry may use specificenvironments that showcase tourist spots, cultural events, exhibitions,amusement parks, natural and historical sites and other places ofinterest to the tourist. The above examples have been mentioned asexemplary embodiments to demonstrate how advertisers can take advantageof the environment module 56 for brand/product advertising purposes.

The entertainment module 66 encompasses activities that include the userbeing able to interact and manipulate their model by animating it toperform different activities such as singing, dancing, etc and using itto participate in gaming and augmented reality environments and otheractivities. Some features associated with the entertainment module 66have already been discussed in the context of the environment module 56.These include the ability of the user to animate the virtual model'smovements, actions, expressions and dialogue; the facility to use themodel in creating music videos, movies, portraits; interacting via themodel with different users in chat sessions, games, shopping trips etc.;and other means by which the user may interact with the virtual model orengage it in virtual activities. Additionally, the entertainment module66 features the user model or another virtual character on the user'sprofile page as an ‘information avatar’ to provide news updates, fashionupdates, information in the form of RSS feeds, news and other feeds andother information that is of interest to the user or that the user hassubscribed to. The character model may supply this information invarious ways, either through speech, or by directing to the appropriatecontent on the page or by displaying appropriate content at the requestof the user, all of which are given as exemplary embodiments. The mainpurpose of using the virtual model to provide information feeds andupdates of interest to the user is to make the process more ‘human’,interactive and to provide an alternative to simple text and imageinformation and feed content. Further to this, the ‘information avatar’or ‘personal assistant’ can incorporate weather information and latestfashion news and trends, as an exemplary embodiment, to suggest apparelto wear to the user. Information from the media agency servers 25 andentertainment servers 23 is used to keep the content reported and usedby the ‘information avatar’ updated. Users will be able to interact witheach other using creative virtual tools. An example includes interactivevirtual gifts. These gifts may embody virtual manifestations of realgifts and cards. Users may have the option to virtually wrap theirpresents using containers, wrapping and decoration of their choice. Theymay also set the time that the virtual gift automatically opens or isallowed to be opened by the gift-receiver. Exemplary embodiments ofgifts include pop-up cards and gifts; gifts withtext/voice/audio/video/animated messages or coupons and other surprises;gifts that grow or change over time. An example of a gift that changesover time constitutes a tree or a plant that is still a seedling or ababy plant when it is gifted and is displayed on the gift-receiver'shome page for example. Over fixed time intervals, this plant/treeanimation would change to reflect virtual ‘growth’ until the plant/treeis fully grown at a specified endpoint. The type of plant/tree may be asurprise and may be revealed when the plant/tree is fully grown at theend of the specified period. There may be a surprise message or anothervirtual surprise/gift that is displayed/revealed to the user when theplant/tree reaches the endpoint of the growth/change interval. Giftsthat change over time may include other objects and are not necessarilyrestricted to the examples above.

The server application 22 also has associated with it a data store 70.The server application 22 has access to the data store 70 that isresident upon the portal server 20 or associated with the portal server20. The data store 70 is a static storage medium that is used to recordinformation associated with the system 10. The data store 70 isillustrated in further detail with respect to FIG. 4.

Reference is now made to FIG. 4 where the components of the data store70 are shown in a block diagram in an exemplary embodiment. Thecomponents of the data store 70 shown here are shown for purposes ofexample, as the data store 70 may have associated with it one or moredatabases. The databases that are described herein as associated withthe data store are described for purposes of example, as the variousdatabases that have been described may be further partitioned into oneor more databases, or may be combined with the data records associatedwith other databases.

The data store 70 in an exemplary embodiment comprises a user database80, an apparel database 82, a 3-D model database 84, and an environmentdatabase 86. The user database 80 in an exemplary embodiment is used torecord and store information regarding a user of the system 10. Suchinformation includes, but is not limited to a user's access login andpassword that is associated with the system 10. A user's profileinformation is also stored in the user database 80 which includes, age,profession, personal information, and user's physical measurements thathave been specified by the user, images provided by the user, a user'shistory, information associated with a user's use of the system. Auser's history information may include, but is not limited to, thefrequency of their use of the system, the time and season they makepurchases, the items they have purchased, the retailers from whom theitems were purchased, and information regarding the various items.Information regarding the various items may include, but is not limitedto, the colour, style and description of the items. The apparel database82 stores information regarding the various items of apparel that areavailable through the system 10. The 3-D model database 86 storespredetermined 3-D models and parts of various 3-D models that arerepresentative of various body types. The 3-D models are used to specifythe user model that is associated with the user. The environmentdatabase 86 stores the various environments that are provided by thesystem 10 and that may be uploaded by users as described below.

Reference is now made to FIG. 5, where a flowchart illustrating thesteps of an access method 100 is shown in an exemplary embodiment.Access method 100 is engaged by the user when the user first logs intothe system 10. The access method 100 describes the various options thatare available to the user upon first accessing the system. Method 100begins at step 101, where the user accesses the system 10 by logginginto the system 10. Users can also browse the system withoutauthentication as a guest. Guests have access to limited content. Asdescribed above in an exemplary embodiment, the system 10 is accessiblethrough the Internet. As the system 10 is accessible through theInternet, the user accesses the system by entering the URL associatedwith the system 10. Each user of the system 10 has a login and passwordthat is used to access the system 10. Upon successful validation as anauthorized user, method 100 proceeds to step 102, where the user ispresented with their respective homepage. The user may be shown theiruser model (if they have previously accessed the system) displayingfeatured items of apparel when they log in. The user is presented with avariety of options upon logging into the system 10. Method 100 proceedsto step 103 if the user has selected to modify their respectiveenvironments associated with the user. At step 103, the user asdescribed in detail below has the ability to modify and alter therespective virtual environments that are associated with the user.Method 100 proceeds to step 104 when the user chooses to manage theirfriends. Users may add other users from within the system 10, and fromexternal community sites as their friends, and may manage theinteraction with their friends. The management of friends in the system10 is explained in further detail below. Method 100 proceeds to step 105when the user wishes to generate or interact with their user model.Method 100 proceeds to step 106 where the user wishes to view items thatmay be purchased. Method 100 proceeds to step 107 where the user mayengage in different collaborative and entertainment activities asdescribed in this document. The steps that have been described herein,have been provided for purposes of example, as various additional andalternative steps may be associated with a user's accessing of theirrespective home page.

Reference is now made to FIG. 6A, where the steps of a detailed modelgeneration method 110 are shown in an exemplary embodiment. The modelgeneration method 110 outlines the steps involved in generating the 3-Duser model. Method 110 begins at step 111, at which the user providesdata to the system 10. The data can be provided all at once orincrementally. The data can be provided by the user or by his/herfriends. Friends may grant or deny access to data request and havecontrol over what data is shared. The data provided may include but isnot limited to image(s) and/or video(s) of the face 113 and/or body 114;measurements 115 of the body size including the head as described below;apparel size commonly worn by the user and the preferred apparel size(s)and preferences 116 for style of clothing (such as fitted, baggy,preferred placement of pants (above, below, or on waist), color,European, trendy, sophisticated etc.), brands, etc.; laser scan data(obtained, for example, from a booth at a store equipped with a laserscanner), meshes (for instance, those corresponding to impressions ofthe ear or foot), outlines of body parts (for instance, those of thehands and feet), mould scans, mocap data, magnetic resonance imaging(MRI), ultrasound, positron emission tomography (PET), and computedtomography (CT) data 117; and other data 118 such as correspondencebetween feature points on the 3D model's surface and the 2D imagessupplied by the user (for example the location of the feature points onthe face as shown in FIG. 11), references to anatomical landmarks on theuser supplied data, and user specific info such as the age or age group,gender, ethnicity, size, skin tone, weight of the user. User data may beimported from other sources such as social-networking sites or thevirtual operating system described later in this document. (Suchimporting of data also applies to the other portals discussed in thisdocument).

The input to the method 110 includes prior information 112 including,but not limited to, annotated 3D surface models of humans that includeinformation such as anatomical landmarks, age, gender, ethnicity, size,etc.; anatomical information, for instance, probability densities offace and body proportions across gender, age groups, ethnic backgrounds,etc.; prior knowledge on the nature of the input data such asshape-space priors (SSPs) (described below), priors on measurements,priors on acceptable apparel sizes, priors on feature pointcorrespondence; sequencing of steps for various action factors(described below), etc. The prior information 112 includes data storedin the data store 70. The prior information 112 is also used todetermine “surprise” as described later in this document.

Based on the information provided at step 111 or data from 113-118,system 10 makes recommendations to the user on stores, brands, apparelas well as provides fit information, as described previously. As usersbrowse apparel, the system informs the user about how well an apparelfits, if the apparel is available in a given user's size and thespecific size in the apparel that best fits the user. In suggesting fitinformation, the system takes into account user fit preferences, forexample a user's preference for loose fit clothing. The system maysuggest whether apparel suits a particular user based on the user'sstyle preferences. In exemplary embodiment, there may be a “your style”field that gives an apparel a score in terms of style preferred by theuser. In another exemplary embodiment, the system may recommend a listof items to the user ordered according to user preferences. Forinstance, a user may prefer collar shirts over V-necks. Furthermore, theuser may not like turtlenecks at all. When this user browses a storecollection with different shirt styles, the system may present the shirtstyles to the user in an ordered list such that the collar shirts areplaced above the V-neck shirts and the turtlenecks are placed towardsthe bottom of the ordered list, so that the user has an easier timesorting out and choosing styles that suit their taste and preferencesfrom the store collection.

In another exemplary embodiment, the system may combine stylepreferences as specified the user, and/or user style based on buyingpatterns of user and/or other users' ratings of apparel, and/or fashionconsultant ratings and/or apparel popularity (assessed according to thenumber of the particular apparel item purchased for example). Anycombination of the above information may be used to calculate the “stylescore” or “style factor” or “style quotient” of a particular item(algorithm providing the score is referred to as “style calculator). Inexemplary embodiment, a user may select the information that the systemshould use in calculating the style factor of a particular item. Theuser may inquire about the style score of any particular item in orderto guide their shopping decision. The system may use the scorescalculated by the style calculator in order to provide apparelrecommendations; style ratings of products and apparel items;user-customized catalogues and lists of products that are ordered andsorted according to an individual's preferences and/or popularity ofapparel items.

Given apparel size, the system can inform a user of the bodymeasurements/dimensions required to fit apparel of the specified size.Alternatively, given a user's body measurements, the system can informthe user of the apparel size that would fit in a given brand ormake/manufacturer. Further, the system can suggest sizes to the user inrelated apparel. In exemplary embodiment, if a user is browsing jacketsin a store and the system has information about the shirt size of theuser, then based on the user's shirt size, the system can suggest theappropriate jacket sizes for the user. In an exemplary embodiment, thesystem can provide fit information to the user using a referencingsystem that involves using as reference a database containing apparel ofeach type and in each size (based on the standardized sizing system).Body measurements specified by a user are used by the system to estimateand suggest apparel size that best meets the user's fit needs (‘fit’information incorporates user preferences as well such as preference forcomfort, loose or exact fit etc.). The reference apparel size databaseis also used to suggest size in any of the different types of apparelsuch as jackets or coats or jeans or dress pants etc. In anotherexemplary embodiment of providing fit information using the referenceapparel database, a user may be looking for dress pants, for instance,and the system may only know the user's apparel size in jeans and notthe user's body measurements. In this case, in exemplary embodiment, thesystem compares jeans in the user's size from the reference appareldatabase with dress pants the user is interested in trying/buying, andby incorporating any additional user fit preferences, the systemsuggests dress pants that would best fit the user i.e., are compatiblewith the user's fit requirements. Fit information may specify anuncertainty along with fit information in order to account for, inexemplary embodiment, any differences that may arise in size/fit as aresult of brand differences and/or apparel material properties and/ornon-standardized apparel size and/or subjectivity in user preferencesand/or inherent system uncertainty, if any exists. In exemplaryembodiment, the system informs a user, who prefers exact fit in shirts,that a shirt the user is interested in purchasing, and which is a newpolyester material with a different composition of materials and thatstretches more as a result, fits with ±5% uncertainty. This is due tothe fact that the stretch may or may not result in an exact fit and maybe slightly loose or may be exact. Since the material is new and thesystem may not have information on its material properties and how sucha material would fit, it cannot provide an absolute accurate assessmentof the fit. It instead uses material information that is close to thenew material in order to assess fit, and expresses the uncertainty infit information. Fit information is communicated to the user, inexemplary embodiment, via text, speech or visually (images, video,animation for example) or any combination thereof. An API (ApplicationProgramming Interface) would be open to vendors on the retail server orportal server on system 10 so that vendors can design and make availableapplications to users of system 10. These applications may include, inexemplary embodiment, widgets/applications that provide fit informationspecific to their brands and products to users; store locaterapplications etc. In an exemplary embodiment, an application that letsvendors provide fit information works simply by looking up in a databaseor using a classifier such as Naïve Bayes [7-9] or k-nearest neighbours(KNN) [9, 10]. For example, an application may state whether a garmentsthat a user(s) is browsing from a catalog fits the user(s). In exemplaryembodiments: (1) Database. The application can look up the user's sizeand the manufacturer of the clothing in a database to find the size(s)corresponding to the given manufacturer that fits the user. If the itemcurrently being viewed is available in the user's size, the item ismarked as such. The database can be populated with such information apriori and the application can add to the database as more informationbecomes available. (2) Naïve Bayes. The a posteriori probability of anapparel size (as) fitting a user given the user's body size (us)information and the manufacturer of the apparel (m) can be computerusing the Bayes rule, This can be expressed as the product of theprobability of the user's size (us) given the apparel size (as) and themanufacturer (m) of the apparel, and that of the prior probability ofthe apparel size given the manufacturer, divided by the jointprobability of the user's size apparel size given the manufacturer (i.e.p(as|us,m)=p(us|as,m)p(as|m)/p(us,as|m)). The prior probabilities can belearnt by building histograms from sufficiently large data andnormalizing them so that the probability density sums to one. The usermay be presented with items that fit the user, or the apparel sizes thatfit the user may be compared with the item that the user is currentlyviewing and if the item that is being viewed belongs to the apparelsizes that fit the user, a check mark or a “fits me” indication may bemade next to the item. (3) KNN. Information on the body size (forexample, measurements of various parts of the body), apparel size fordifferent manufacturers for both males and females, and (optionally)other factors such as age are stored in a database for a sufficientlylarge number of people. Each of these prices of information (i.e. bodysize, apparel size) is multiplied by a weight (to avoid biases). Given anew body size, the closest exemplar is found by computing the Euclideandistance between the given body size (multiplied by the associatedweights for each measurement) and those in the database, The majorityvote of the output value (i.e. the corresponding field of interest inthe database, for example, the apparel size corresponding to the bodymeasurements) of the k-nearest neighbours (where k is typically taken tobe an odd number) is taken to be the most reasonable output. This outputvalue is then divided by the corresponding weigh (weight can take thevalue 1 also). This could also be used in any other combination ofinputs and outputs. For example, the input could be the apparel size fora given manufacturer and the output could be the body sizes that fitthis apparel. In an exemplary embodiment, when browsing for products,given the user's body size (which may be stored in a repository) and themanufacturer whose items the user is currently looking at, the apparelsizes that fit the user may be computed and the user may be presentedwith the available sizes for the user. The user can also filter catalogsto show only items that fit the user or correspond to the user'spreferences.

Based on a user's apparel size, the system can point out to the user ifa product is available in the user's size as the user is browsingproducts or selecting products to view. The system may also point outthe appropriate size of the user in a different sizing scheme, forexample, in the sizing scheme of a different country (US, EUR, UK etc.).In suggesting appropriate sizes to user in products that may varyaccording to brand, country, and other criteria, the system also takesinto account user fit preferences. For instance, a user may want clothesto be a few inches looser than his/her actual fit size. In an exemplaryembodiment, the system would add the leeway margin, as specified by theuser, to the user's exact fit apparel size in order to find the desiredfit for the user.

Method 110 begins at the preprocessing step 119 at which it preprocessesthe user data 111 using prior knowledge 112 to determine the appropriatecombination of modules 120, 123, 124, 125, and 126 to invoke. Method 110then invokes and passes the appropriate user data and prior knowledge toan appropriate combination of the following modules: image/videoanalysis module 120, measurements analysis module 123, apparel sizeanalysis module 124, mesh analysis module 125, and a generic module 126as described in detail below. These modules 120, 123,124, and 125attempt to construct the relevant regions of the user model based on theinput provided. At the information fusion step 127, the data produced bythe modules 120, 123,124, 125 and 126 is fused. Method 110 theninstantiates a preliminary model at step 128, optimizes it at the modeloptimization step 129, and details it at step 130. Method 110, thenpresents the user with a constructed model at step 131 for usermodifications, if any. The constructed model and the user changes arepassed on to a learning module 132, the output of which is used toupdate the prior knowledge in order to improve the model constructionmethod 110. As method 110 proceeds, its intermediary progress is shownto the user. At any point during the model construction method 110, theuser is allowed to correct the method. In an exemplary embodiment, thisis done by displaying the model at the intermediately steps along withthe parameters involved and allowing the user to set the values of theseparameters though an intuitive interface. At the conclusion of method110, a user model is generated. Each of the steps of method 110 isdescribed in further detail below.

Measurements 115 provided as input to the method 110 include, in anexemplary embodiment, measurements with respect to anatomical landmarks,for example, the circumference of the head and neck, distance fromtrichion to tip of nose, distance from the tip of the nose to the mentalprotuberance, width of an eye, length of the region between the lateralclavicle region to anterior superior iliac spine, circumference of thethorax, waist, wrist circumference, thigh circumference, shin length,circumference of digits on right and left hands, thoracic musclecontent, abdominal fat content, measurements of the pelvis, measurementsof the feet, weight, height, default posture (involving measurementssuch as elevation of right and left shoulders, stance (upper and lowerlimbs, neck, seat, waist, etc.), humping, etc.). Apparelsize/preferences 116 include, in an exemplary embodiment, clothing sizesuch as dress size (eg. 14, 8, etc.), hat size, shoe size, collar size,length of jacket, trouser inseam, skirt length etc., including anindication of whether measurements represent an exact size or include apreferred margin or are taken over clothes. The specific measurementsdiffer for males and females reflecting the anatomical differencebetween the genders and differences in clothing. For instance, in thecase of females, measurements may include a more elaborate measurementof the upper thorax involving measurements such as those of the largestcircumference of the thorax covering the bust, shoulder to bust length,bust to bust length etc. On the other hand, in the case of males, owingto lower curvature, fewer measurements of the chest may be required.Similarly, for the case of clothing, women may provide, for instance,the length of a skirt, while men may provide a tie size. Similarly,children and infants are measured accordingly. The availability ofinformation on anatomical landmarks makes it possible to deriveanatomically accurate models and communicate fit information to the useras described below. Strict anatomical accuracy is not guaranteed whennot desired by the user or not possible, for example, under stringentcomputational resources. A printable tape measure is provided to theuser as a download to ease the process of measuring. Image(s) and/orvideo(s) of the face 113 and/or body 114 provided to the system can alsobe imported from other sources and can also be exported to otherdestinations. In an exemplary embodiment, the method 110 may use imagesthat the user has uploaded to social networking sites such as Facebookor Myspace or image sharing sites such as Flickr.

The method 110 can work with any subset of the data provided in 111,exemplary embodiments of which are described below. The method 110 isrobust to incomplete data and missing information. All or part of theinformation requested may be provided by the user i.e. the informationprovided by the user is optional. In the absence of information, priorknowledge in the form of symmetry, interpolation and other fill-inmethods, etc are used as described below. In the extreme case of limiteduser data, the method 110 instantiates, in an exemplary embodiment, ageneric model which could be based on an average model or a celebritymodel. Depending on factors such as the information provided by theuser(s), computational power of the client platform, shader support onclient machine, browser version, platform information, pluginsinstalled, server load, bandwidth, storage, user's preferences (eg.photorealistic model or a version of nonphotorealistic rendering (NPR))etc., the method 110 proceeds accordingly as described below. Thesefactors are herein referred to as action factors. Depending on theaction factors, a 3D model of appropriate complexity is developed. Whena highly complex (a higher order approximation with a higher poly count)model is generated, a downsampled version (a lower poly count model) isalso created and stored. This lower poly count model is then used forphysical simulations in order to reduce the processing time while thehigher poly count model is used for visualization. This allows plausiblemotion and an appealing visualization. Goodness of fit information forapparel is computed using the higher poly count model unless limited bythe action factors.

Method 110, at the preprocessing step 119 at which it preprocesses theuser input data using prior knowledge to determine which of the modules120, 123,124, 125 and 126 to invoke; depending on the input provided andthe action factors, an appropriate combination of modules 120, 123,124,125 and 126 is invoked. The method 110 attempts to construct the mostaccurate model based on the data for the given action factors. Theaccuracy of a model constructed using each of the modules 120, 123, 124,125 and 126 is available as prior knowledge 112, and is used todetermine the appropriate combination of modules 120, 123, 124, 125 and126 to invoke. In an exemplary embodiment where the client platform iscomputationally advanced (modern hardware, latest browser version,shader support, etc.), if only images of the face and body are providedby the user, only the image/video analysis module 120 is invoked; ifonly body measurements are provided, only the measurements analysismodule 123 is invoked; if only apparel size information is provided,only the apparel size analysis module 124 is invoked; if only a fullbody laser scan is provided, only the mesh analysis module is invoked;if only apparel size information and an image of the face is provided,only the apparel size analysis module 124 and the images/videos analysismodule, more specifically the head analysis module 121, are invoked; ifonly an image of the face is provided, only the generic module 126 andthe images/videos analysis module, more specifically the head analysismodule 121, are invoked; if an image of the face, body measurements anda laser scan of the foot is provided the image/videos analysis module,more specifically the head analysis module 121, the measurementsanalysis module and the mesh analysis modules are invoked and so on. Forregions of the body, for which information is unavailable, the genericmodule is invoked. In the extreme case of no user information or verylimited computational resources, only the generic module 126 is invoked.Other data 118 such as age and gender, if provided, and prior knowledgeis available to each to the modules 120, 123,124, 125 and 126 to assistin the model construction process. Parameters may be shared between themodules 120, 123,124, 125 and 126. Each of the modules 120, 123,124, 125and 126 are described in detail next.

Reference is now made to the images/videos analysis module 120 in FIG.6A. This module consists of a head analysis module 121 and a bodyanalysis module 122, in an exemplary embodiment. The head analysismodule 121 and the body analysis module 122 construct a 3-D model of theuser's head and body, respectively, based on the image(s) and video(s)provided. The head analysis module 121 and the body analysis module 122may work in parallel and influence each other. The head analysis module121 and the body analysis module 122 are described in detail below.

Reference is now made to FIG. 6B where the steps of the modelconstruction process of the images/videos analysis module 120 areoutlined in an exemplary embodiment. After receiving image and or videofile(s), this module extracts information on the user's physicalattributes at step 137 and generates a three-dimensional model at step138. A detailed description of this process is provided below.

Reference is now made to FIG. 6C where it is shown, in an exemplaryembodiment, that the steps of the model construction process in theimage/video analysis module are handled separately for the user's faceand the body. The head analysis module 121 produces a model of theuser's head while the body analysis module 122 produces a model of theuser's body. These models are then merged at the head-body fusion step.A detailed description of this process is provided below.

Reference is now made to FIG. 6D, wherein a detailed description of themodel generation process of the images/videos analysis module 120 forsteps 121 and 122 is provided in an exemplary embodiment. The steps ofthe model construction are first described in the context of the headanalysis module 121. The body analysis module 122 proceeds in a similarfashion. Once invoked by method 110, the module 120 after receivingimage(s) and/or videos and prior knowledge, first sorts the data intoimages and videos at step 139, based on the file extension, file header,or user tag in an exemplary embodiment. If only image(s) are present,the method proceeds to the preprocessing step 141. If only video(s) arepresent, the method first extracts images from the video thatapproximately represent a front view of the face and/or a side view ofthe face, if available and proceeds to the processing step 141. This isdone in an exemplary embodiment using a technique similar to that usedin [11]. In another exemplary embodiment, a 3D model of the face isconstructed using a technique similar to that in [12]. If a combinationof videos and images are present and the resolution of the image(s) ishigher than that of the video, the method proceeds to the preprocessingstep 141 using the higher resolution images. If a low resolution videois present, for example a video captured using a cell phone, highresolution images are first generated and then the method proceeds tothe processing step 141. This can be done, in an exemplary embodiment,using a technique similar to that used in [13]. Stereo images and/orvideos can also be processed. In an exemplary embodiment, this can bedone using a technique similar to [14].

Reference is now made to the preprocess step 141 in FIG. 6D of theimage/video analysis module 120, wherein the image(s) are preprocessed.This involves, in an exemplary embodiment, resizing, scaling,de-noising, etc., if necessary to bring the images to a canonical form.An approximate region containing the face region in the images isidentified at this step. This is done, in an exemplary embodiment, usinga rotationally invariant neural network. In another exemplaryembodiment, this can be done using support vector machines (SVMs) in amanner similar to that described in [15]. The location(s) of the face(s)in the image(s) and associated parameters (eg. approximate facial pose,scale, etc.), and a probability density over the image space identifyingthe foreground (face regions) and the background are then passed to thenext step. In an exemplary embodiment, this density is defined as aGaussian about the location of the face. Facial pose is defined as the3D orientation of a person's face in 3D space. It can be parameterized,in an exemplary embodiment, by the orientation of the line joining theeyes and the two angles between the facial triangle (formed by the eyesand nose) and the image plane. The scale of the image is computed, in anexemplary embodiment, using (i) the measurement of a reference region asmarked by the user, if available, or (ii) the size of a common object(eg. a highlighter) in the image at approximately the same depth as theperson in the image, if available, or (ii) the measured size of a knownobject (eg. a checkered pattern) held by the user in the image. Ifmultiple faces are detected in a single image, the user may be askedwhich face the user would like a model created for or a model may becreated for each face in the image allowing the user to decide whichones to store and which ones to delete. The method 110 then proceeds tostep 148, where the global appearance is analyzed, and step 142, wherethe local features of the head are analyzed. The global appearanceanalysis step 148 involves, in an exemplary embodiment, projecting theforeground on a manifold constructed, for example, using principalcomponent analysis (PCA), probabilistic principal component analysis(PPCA), 2D PCA, Gaussian Process Latent Variable Models GPLVM, orindependent component analysis (ICA). This manifold may be parameterizedby global factors such as age, gender, pose, illumination, ethnicity,mood, weight, expression, etc. The coefficients corresponding to theprojection are used to produce a likelihood of observing the imagesgiven a face model. In an exemplary embodiment, this is given by aGaussian distribution centered at the coefficients corresponding to theprojection. The estimated parameters from the previous step are updatedusing Bayes rule and the likelihood determined at this step. Theposterior global parameters thus computed serve as priors at step 142.Depending on the action factors, the method 110 segments the face intovarious anatomical regions (steps 143-146), projects these regions ontolocal manifolds (at steps 149 and 150) to generate local 3D surfaces,fuses these local 3D surfaces and post processes the resulting headsurface (steps 151 and 152), optimizes the model 153 and adds detail tothe model 154. These steps are described in detail below.

The method 110 at step 142 identifies various anatomical regions of theface in the image and uses this information to construct a 3D surface ofthe head. This is done, in an exemplary embodiment, using shape spacepriors (SSPs). SSPs are defined here as a probability distribution onthe shape of the regions of an object (in this context a face), therelative positions of the different regions of the object, the textureof each of these regions, etc. SSPs define a prior on where to expectthe different regions of the object. SSPs are constructed here based onanatomical data. In an exemplary embodiment, an SSP is constructed thatdefines the relative locations, orientations, and shapes of the eyes,nose, mouth, ears, chin and hair in the images. Using priors from step148 and SSPs on the face, the method 110 at step 143 extracts basicprimitives from the images such as intensity, color, texture, etc. Themethod 110 at step 2326, to aid in segmentation of facial features,extracts more complex primitives such as the outlines of various partsof the face and proportions of various parts of the face usingmorphological filters, active contours, level sets, Active Shape Models(ASMs) (for example, [16]), or a Snakes approach [17], in an exemplaryembodiment. As an example, the active contours algorithm deforms acontour to lock onto objects or boundaries of interest within an imageusing energy minimization as the principle of operation. The contourpoints iteratively approach the object boundary in order to reach aminima in energy levels. There are two energy components to the overallenergy equation of an active surface. The ‘internal’ energy component isdependent on the shape of the contour. This component represents thefacets acting on the contour surface and constraining it to be smooth.The ‘external’ energy component is dependent on the image propertiessuch as the gradient, properties that draw the contour surface to thetarget boundary/object. At step 146, the outputs of steps 143 and 144which define likelihood functions are used together with SSPs, in anexemplary embodiment using Bayes rule, to segment the regions of thehead, helmet, eyes, eyebrows, nose, mouth, etc. in the image(s). Ahelmet is defined here as the outer 3D surface of the head including thechin, and cheeks but excluding the eyes, nose, mouth and hair. Theresult is a set of hypotheses that provide a segmentation of variousparts of the head along with a confidence measure for each segmentation.(Segmentation refers to the sectioning out of specific objects fromother objects within an image or video frame. In an exemplaryembodiment, an outline that conforms to the object perimeter isgenerated to localize the object of interest and segregate it from otherobjects in the same frame). The confidence measure, in an exemplaryembodiment, is defined as the maximum value of the probability densityfunction, at the segmented part's location. If the confidence measure isnot above a certain threshold (in certain challenging cases eg. partialocclusion, bad lighting, etc.), other methods are invoked at theadvanced primitive extraction step 145. (For example methods based ondepth from focus, structure from motion, structure from shading,specularity, silhouette, etc.; techniques similar to [18], [19], [20],[21] and [22]). In an exemplary embodiment, this is done by selecting amethod in a probabilistic fashion by sampling for a method from aproposal density (such as the one shown in FIG. 6I). For example, if theface of the user is in a shadow region, a proposal density is selectedthat gives the probability of successfully segmenting the parts of aface under such lighting conditions for each method available. From thisdensity a method is sampled and used to segment the facial features andprovide a confidence measure of the resulting segmentation. If theupdated confidence is still below the acceptable threshold, theprobability density is sampled for another method and the process isrepeated until either the confidence measure is over the threshold orthe maximum number of iterations is reached at which point the methodasks for user assistance in identifying the facial features.

As each of the features or parts of the face is successfully segmented,a graphical model is built that predicts the location of the otherremaining features or parts of the face. This is done using SSPs tobuild a graphical model (for eg. a Bayes Net). Reference is made to FIG.6E, where a graphical model is shown in an exemplary embodiment, and toFIG. 6F, where the corresponding predicted densities are shown in imagecoordinates. The connections between the nodes can be built in parallel.As the method progresses, the prior on the location from the previoustime step is used together with the observation from the image (resultof applying a segmentation method mentioned above), to update theprobability of the part that is being segmented and the parts that havebeen segmented, and to predict the locations of the remaining partsusing sequential Bayesian estimation. This is done simultaneously formore than one part. For example, if the location of the second eye isobserved and updated, it can be used to predict the location of thenose, mouth and the eyebrow over the second eye as shown in FIG. 6E. Asimplified walkthrough of the sequential Bayesian estimation forsegmenting the regions of the face is shown in FIG. 6F.

Simultaneously with steps 143-145, the pose of the face is determined.In an exemplary embodiment, on identification of specific facialfeatures such as the eyes and mouth, an isosceles triangle connectingthese features is identified. The angle of facial orientation is thendetermined by computing the angle between this isosceles triangle andthe image plane. The pose thus computed also serves as a parameter atthe classification step 151. The segmentation methods used are designedto segment the parts of the head at smooth boundaries. Next, parameterscorresponding to these parts such as pose, lighting, gender, age, race,height, weight, mood, face proportions, texture etc. are computed. In anexemplary embodiment, this is done as follows: once a majority of theparts of the head are identified, they are projected onto acorresponding manifold in feature space (eg. edge space). In anexemplary embodiment, a manifold exists for each part of the face. Thesemanifolds are built by projecting the 3D surface corresponding to a partof the face onto an image plane (perspective projection) for a largenumber of parts (corresponding to different poses, lighting conditions,gender, age, race, height, weight, mood, face proportions, etc.),applying a feature filter (eg. a Canny edge detector) at step 149 toconvert to a feature space (eg. edge space, color space, texture space,etc.), and then applying a dimensionality reduction technique such asprincipal component analysis (PCA), probabilistic principal componentanalysis (PPCA), 2D PCA, Gaussian Process Latent Variable Models GPLVM,or independent component analysis (ICA). Since the manifolds areparameterized by pose, lighting, gender, age, race, height, weight,mood, face proportions, texture etc., projecting a given segmented partof the head onto the manifold allows recovery of these parameters (forexample [23]). These parameters are then passed onto a classifier (atstep 151), in an exemplary embodiment, a Naïve Bayes classifier, asupport vector machine (SVM), or a Gaussian Process classifier, tooutput the most plausible 3D surface given the parameters. In anexemplary embodiment, if a particular parameter is already supplied aspart of 118, for eg. the gender of the user, then it is used directlywith the classifier and the corresponding computation is skipped (eg.estimation of gender). Teeth reconstruction is also handled similarly.The teeth that are constructed are representative of those in the imageprovided including the color and orientation of teeth. This is neededlater for animation and other purposes such as to show virtually resultsof dental corrections, whitening products, braces, invisalines, etc.Hair are also handled similarly. In this case, the manifold isadditionally parameterized by the 3D curvature, length, specularity,color, 3D arrangement, etc. In an exemplary embodiment, a helical modelis used as the underlying representation for a hair strand. In anexemplary embodiment hair can be modeled from image(s) using techniquessimilar to [24-26]. If, however, the action factors do not allow arepresentation of the teeth, ears and hair exactly as in the image, lesscomplex precomputed models are used. Once 3D surface exemplars forvarious parts of the head (for example, a helmet defined below, eyes,nose, mouth, etc.) are identified as outputs of the classifier, at step152 a new model is instantiated by instantiating a copy of theidentified exemplar surfaces. Since the instantiated surfaces areparametric by construction, these parametric models are modifiedslightly (within allowed limits), if necessary, to represent parametersas extracted from the image(s) wherever possible at the optimizationstep 153. The exemplars that are used with the classifier are riggedmodels and thus enable easy modifications. In an exemplary embodiment,the size of the skeletal structures and the weight of the nodes aremodified to match the extracted parameters. The rigged models also allowuser modifications (as described with reference to FIG. 29B) andfacilitate animations. At the postprocessing step 154, the 3D surfacesgenerated at step 153 are merged. The boundaries of the 3D surfacescorresponding to the parts of the face are merged and smoothed usingtechniques similar to those used at the head-body fusion step 155 (FIG.6C). Symmetry is used to complete occluded or hidden parts. For example,if the user's hair are partially occluding one side of the face,symmetry is used to complete the missing part. If not enough informationis available, the most likely surface and texture are substituted. Forexample. if the user's teeth not visible owing to the mouth beingclosed, the most likely set of teeth, given the parameters correspondingto the user. In an exemplary embodiment, the most likely surface andtexture are computed using a classifier such as Naïve Bayes, while theplacement is computed using SSPs and Bayesian inference. As an alternateembodiment, 3D surfaces of the entire head for different combinations ofconstituent part parameters are maintained and an appropriate model isinstantiated at step 152 based on the output of the classification step151. At the conclusion of the postprocessing step 154, a preliminary 3Dmodel of the user's head is available which is passed onto the head-bodyfusion step 155. As mentioned earlier, the body analysis module 122proceeds similar to the head analysis module 121, where instead ofextracting parameters of parts of the face, parameters of the variousbody parts (excluding the head) are extracted from the image(s) and/orvideos. In an exemplary embodiment, the local feature analysis step 142for the body analysis module 122 involves individually analyzing theupper limbs, the lower limbs, the thorax, the abdomen, and the pelvis.In an exemplary embodiment, the location of the body in the image andits pose is identified at the preprocessing step 141 using a techniquesimilar to that used in [27]. At the conclusion, of the postprocessingstep 154 of the body analysis module 122, a preliminary 3D model of theuser's body is generated which is passed onto the head-body fusion step155.

At the head-body fusion step 155, the head model estimate and the bodymodel estimate are merged using smoothness assumptions at theboundaries, if necessary. In an exemplary embodiment this isaccomplished by treating the regions at the boundaries as B-splines andintroducing a new set of B-splines to interconnect the two regions to bemerged (analogous to using sutures) and shrinking the introduced linksuntil the boundary points are sufficiently close. A 1-D example is shownin FIG. 6G. Alternatively, the boundaries at the neck region may beapproximated as being pseudo-circular and the radii of the body model'sneck region and the head model's neck region can be matched. This mayinvolve introducing a small neck region with interpolated radius values.Other methods such as the one proposed in [28] could also be used. Thechoice of the method used for fusion depends, in an exemplaryembodiment, on the action factors. For instance, if limited data isprovided by the user leading to a relatively coarse approximation to theuser, the pseudo-circular approximation method mentioned above is used.As another example, a particular version of an NPR model desired by theuser may not require sophisticated model for which the pseudo-circularapproximation method mentioned above is used. The output of thehead-body fusion step 155 is passed onto the information fusion step127.

Reference is now made to the measurements analysis module 123 thatprocesses the measurements provided by the user in order to construct auser model or part thereof. These measurements include the various headand body measurements 115 provided by the user. The measurements 115provided are used to estimate any missing measurements based onanatomical and anthropometric data, and data on plastic surgeryavailable as part of the prior knowledge 112. As an example of theconstruction of a head model, given the width, x, of one of the user'seyes, the proportions of the remaining parts of the head are generatedbased on anthropometric data as follows: the diameter of the head, alongthe eyes and the ears is taken to be 5×, the distance from the trichionto the menton is taken to be 6×. If the user's ethnicity is known, thenthe shape is appropriately adjusted based on anthropometric data. Forexample, the shape of an average Asian head as seen from above iscircular while that of an average Caucasian is elliptical. Thisinformation is then passed to a classifier to output the most plausible3D surface of the head given the parameters. Measurements of the bodyare used to instantiate a model corresponding to these measurements froma generative model. A generative model is available as part of the priorknowledge 112 and is constructed, in an exemplary embodiment, usinganthropometric data. In an exemplary embodiment, this is done usingtechniques similar to those used in [29, 30]. If a very limited numberof measurements are available in addition to images, they are passedonto the classifier at step 151 and the extraction of the correspondingmeasurement from the image(s) or video(s) is skipped, in an exemplaryembodiment. The output of the measurements analysis module is passedonto the information fusion step 127.

Reference is now made to the apparel size analysis module 124 in FIG. 6Athat processes the apparel size/preferences 116 provided by the user inorder to construct a user model or part thereof. Prior knowledge 112includes an association of an average 3D model with size data forshirts, dresses, trousers, skirts, etc. For example, there is an average3D model of the upper body of a male associated with a men's shirtcollar size of 42 and similarly a model of the lower body for a trouserwaist size of 32 and a length of 32, or a hat size of 40 cm, or a shoesize of 11. This can be done, in an exemplary embodiment, by computingthe average of the upper body 3D surface of several models (obtainedfrom rage scans after filtering noise and rigging) of men who haveidentified a collar size of 42 as their preferred shirt size. In anotherexemplary embodiment, the generative models learnt from anthropometricdata, for example as in [29] may have size parameters mapped to apparelsize, thereby giving a generative model that is parameterized by apparelsize. These models are also rigged, in an exemplary embodiment using atechnique similar to that used in [31], to allow animation. Thus, in anexemplary embodiment, a user model can be created from apparel size databy (i) instantiating the corresponding average 3D model for the variousbody parts for which an apparel size is specified, or instantiating thepart of the body corresponding to the apparel using a generative modelparameterized by apparel size, and (ii) merging the 3D surfaces for thevarious body parts using merging techniques similar to those used atstep 155 using most probable generic models for body parts (availablefrom the generic module 126) for which apparel size is not provided. Theoutput of the apparel size analysis module is passed onto theinformation fusion step 127.

Reference is now made to the mesh analysis module 125 in FIG. 6A thatprocesses the laser scan data/meshes/outlines 117 provided by the userin order to construct a user model or part thereof. The steps of themesh analysis module are shown in FIG. 6H in an exemplary embodiment.After receiving user data 111 and prior knowledge 112, once invoked,this module first sorts 156 the data [such as laser scan data, meshes(for instance, those corresponding to impressions of the ear or foot),outlines of body parts (for instance, those of the hands and feet),mocap (motion capture) data, magnetic resonance imaging (MRI),ultrasound, positron emission tomography (PET), and computed tomography(CT) data] to determine the most accurate choice of data to use formodel construction. This is done using knowledge of the accuracy of amodel constructed using each piece of the pieces of data above,available as part of prior knowledge 112, and the quality of the dataprovided such as the poly count of a mesh. The user is also allowed toforce the use of a preferred data, for example mocap data as opposed toa laser scan, for model construction by specifying the reliability ofthe data manually. For meshes, the module 125 then proceeds as follows:The module 125 filters the data at step 157 to remove any noise and tocorrect any holes in the data. This is done, in an exemplary embodiment,using template-based parameterization and hole-filing techniques similarto those used in [29]. At this step, unnecessary information such asmeshes corresponding to background points is also removed. This can bedone, in an exemplary embodiment, by asking the user to mark suchregions through an intuitive user interface. This is followed by thefill-in step 158 at which symmetry is used to complete missing regionssuch as an arm, if any, using symmetry. If mesh or volume data is notavailable for the missing regions, the corresponding regions aregenerated by the generic module 126 and fused at the information fusionstep 127. The model is then rigged at the rigging step 159. Riggingprovides a control skeleton for animations and also for easily modifyingthe body parts of the user's model. The mesh output from step 158 isused with a generic human skeleton and an identification of theorientation of the mesh to automatically rig the mesh. Generic male andfemale versions one for age group 0-8, 8-12, 13-20, 21-30, 31-60, 60+ inan exemplary embodiment are available as part of the prior knowledge112. The orientation of the mesh (i.e which side is up) is obtained fromthe mesh file's header. If unavailable in the header, the orientation ofthe header is obtained by asking the user through an intuitive userinterface. Rigging is done automatically, in an exemplary embodiment,using a technique similar to that used in [31]. It can also be doneusing techniques similar to those used in [32, 33].

For laser scan data, a mesh is first constructed, in an exemplaryembodiment, using a technique similar to that used in [34]. This mesh isthen passed on to the fill-in step 158 and the rigging step 159described above. For mocap data, a model is generated using shapecompletion techniques such as that used in [35], in an exemplaryembodiment. The model thus generated is rigged automatically, in anexemplary embodiment, using a technique similar to that used in [31].For outlines, this module extracts constraints from the outlines andmorphs the mesh to satisfy the constraints. In an exemplary embodiment,this is done as follows: (i) Feature points on the outline correspondingto labeled feature points on the mesh (for example, points over the endsof eyebrows, over the ears, and the occipital lobe) are identified bythe user through a guided interface such as the one shown in FIG. 11.This can also be automated using perceptual grouping and anatomicalknowledge. For example, consider a scenario where a user prints out asheet that has a reference marker from the website and draws an outlineof his/her foot, or takes an image of his/her foot with a penny next tothe foot. Given such an image, the image is first scaled to match theunits of the coordinate system of the 3D mesh using scale informationfrom the reference markers in the image. If a reference marker is notpresent, the image is search for commonly known objects such as ahighlighter or a penny using template matching and the known size ofsuch objects is used to set the scale of the foot outline. Or, the usermay be asked to identify at least one measurement on the foot. Theorientation of the foot is then identified. This is done by applying aCanny edge detector to get the edge locations and the orientations,connecting or grouping edgels (a pixel at which an edge has beenidentified) that have an orientation within a certain threshold, andfinding the longest pair of connected edges. This gives the orientationof the foot. Both ends of the foots are searched to identify the regionof higher frequency content (using a Fourier Transform or simplyprojecting the region at each end onto a slice along the foot andlooking at the resulting histogram) corresponding to the toes. The bigtoe is then identified by comparing the widths of the edges defining thetoes and picking the one corresponding to the greatest width. Similarly,the little toe and the region corresponding to the heel are identifiedand reference points on these regions corresponding to those on the 3Dmeshes are marked which now define a set of constraints. (ii) Thecorresponding reference points are then displaced towards the identifiedreference points from the image using Finite Element Analysis (FEM)techniques such as those used in [36], [37], or as in [38]. Theextracted constraints are also passed onto the other modules 120, 123,124 and 126 and a similar method is applied to ensure that the generatedmodel conforms to the constraints. Such morphing of the mesh to conformto constraints is particularly used, if action factors allow, for partsof the body that cannot be easily approximated by a cylinder such as thehead. Such morphing of the mesh based on constraints provided by theuser such as an outline or an image of their foot or fingers are usefulfor computing goodness of fit information for apparel such as shoes andrings. (For the case of rings, it is also possible to simply measure thecircumference of the ring and let the measurement analysis moduleconstruct the appropriate model). For rings, two roughly orthogonalimages of the fingers with a reference material in the background or anoutline of the fingers on a printable sheet containing a referencemarker could be used and analyzed as above. Or, a users hand can beplaced in front of a webcam with a ref on paper in the background or acomputer screen in the background containing a reference marker. Theadvantage of such an image based constraint extraction is that it allowsmultiple fingers to be captured at once. This is particularly usefulwhen buying, say mittens or gloves or a ring, for a friend as a surprisegift. The user simply needs to take an image(s) of the appropriateregion of his/her friend's body, mark the size of some known object inthe image, for example, the width of the user's face. The moreinformation is provided, the more accurate the user's model becomes. Forexample, for some people, the ring size for the right index finger isdifferent from that of the left hand; images of both hands ensure a moreaccurate goodness-of-fit. Imprints and moulds such as those of the footand ears can be converted to meshes can be done either by laserscanning. It can also be done taking multiple images of the imprints andmoulds and constructing the mesh using structure from focus, structurefrom motion, structure from shading, specularity, etc.; techniquessimilar to those used in [18] and [22]. Medical images and volumes suchas MRI and CT volumes can also be used, if available, to create the usermodel or part thereof. This can be done using techniques similar tothose used in [39, 40].

For images from multiple views of a user with known image acquisitiongeometry, a volume is first created as follows and processed asdescribed above for the case of laser scan data. (i) Each image ispreprocessed and a transform is applied producing a feature space image.For example, a silhouette transform is applied which produces an imagewith a silhouette of the object(s) of interest. This can be done in anexemplary embodiment using a technique similar to that used in [41].(ii) The silhouette is then backprojected. This can be done, in anexemplary embodiment, by summing the contributions from each of thesilhouettes taking into account the geometry provided as shown in FIG.6J. Using the geometry of the image capture (this is usually aperspective projection or can be approximated with an orthographicprojection), rays are traced from pixels on the feature spacetransformed images to voxels (3D pixels) of a volume (a 3D image). Toeach of the voxels along the path of a ray, the value of the pixel inthe feature space transformed image is added. This added value may becorrected for a 1/r² effect (inverse square law of light andelectromagnetic radiation). Once a mesh is created, knowledge of thesilhouette is used to extract the texture of the object of interest andusing image acquisition geometry, the model is textured as described atthe primary model instantiation step 128. It can also be done in thefrequency domain using a technique similar to that described in [42].Instead of using the silhouette above, any other feature space transformcan be used. For images from multiple views of an object(s) with unknownor limited geometry information, the images are processed as describedabove with geometry information extracted from the images as follows:(i) Detect salient features. This is done in an exemplary embodiment byusing statics on regions that are interesting to humans extracted bytracking eye movements. In another exemplary embodiment, it can be doneusing prior knowledge of the parts of the object of interest. Forexample, the eyes, nose and mouth can be identified similar totechniques used at step 121 (ii) Form triangles by connecting thesalient features. For example, the eyes, nose, and mouth of a person inan image may be connected to form a triangle. (iii) Determine image toimage transformations of the corresponding triangles. This can be donein an exemplary embodiment using a technique similar to that used in[43]. These transformations define the image acquisition geometry whichis then processed along with the images to construct a model asdescribed above. In Instead of using triangles other structures ornetwork of structures may be used above. The method described aboveallows construction of a model from arbitrary views of an object orperson taken using an ordinary camera. Planes in the image can also beidentified by detecting lines diminishing towards a vanishing point.This can be used to construct a model of the environment, if desired. Itcan also be used to aid in background subtraction. A technique similarto the one presented in [44] can also be used for the environment. Theoutput of the mesh analysis module is passed onto the information fusionstep 127.

Reference is now made to the generic module 126 in FIG. 6A to constructa user model or part thereof. This module processes other data 118, ifavailable, together with prior knowledge 112 in order to produce ageneric model or part thereof. This module is invoked when there isinsufficient information for constructing a user model or part thereofvia the other modules 120, 123, 124, and 125, or if the action factorsdo not allow the generation of a more accurate model that is conformalto the user through modules 120, 123, 124, and 125. When invoked, theinformation in other data 118 or that provided by the modules 120, 123,124, and 125 is passed onto a classifier similar to that used at step151. In an exemplary embodiment, a Naïve Bayes classifier, a supportvector machine (SVM), or a Gaussian Process classifier is used, tooutput the most plausible 3D surface given the information. If only apart of the model (such as a limb) is required by the other modules 120,123, 124, and 125, then only the required part is generated using theclassifier. If the whole model is required, then the entire user modelis generated using the classifier. In an exemplary embodiment, theclassifier outputs an exemplar that is a rigged model. The riggedexemplar is then modified, if necessary, to better match the user. Forexample, if other data 118 specifies an age of five years and a heightof five feet, and the closest exemplar is a user model corresponding toa five year old that is four and half feet tall, the height of thisexemplar is changed from four and half to five feet by setting theparameters of the rigged user model accordingly. The classifier is builtusing labeled training data. In an exemplary embodiment, this is doneusing rigged 3D surfaces or meshes that have associated with them labelsidentifying the age, gender, weight, height, ethnicity, color, apparelsize etc. of the corresponding 3D surface or mesh. The labeling can bedone manually as it only needs to be done once when building theclassifier. The classifier is stored and available as part of priorknowledge 112. As more and more data becomes available, the classifieris updated at the learning step 132. In essence, the method 110 isconstantly learning and improving its model construction process.

The processed information from the modules 120, 123, 124, 125, and 126,if available, is then fused at the information fusion step 127. At thisstep, merging of the outputs of components of 120, 123, 124, 125, and126 takes place. There is an accuracy associated with the output of themodules 120, 123, 124, 125, and 126 available as part of prior knowledge112. Based on this accuracy components of various parts of the user'smodel are merged. For example, the full body output of the genericmodule 126 may be merged with a high resolution model of the user's footavailable as an output of the mesh analysis module 125. This can bedone, in an exemplary embodiment, using techniques similar to those usedat the head-body fusion step 155. Parts of the skeleton are also joinedat the joint locations. For example, for the above example, the fullbody skeleton is joined with the foot skeleton at the ankle joint. Forregions of the body for which data is unavailable, the output of thegeneric module is used. For regions of the body for which multiplemodels of similar accuracy exist, the corresponding models are merged ina probabilistic framework. For example, the expected value of this 3Dmodel's surface is computed over all pieces of data available as outputsof 120, 123, 124, 125, and 126 to produce an estimate of the 3D model ofthe user's head. In an exemplary embodiment, this is done using Bayesianmodel averaging, committees, boosting and other techniques for combiningmodels may be used.

At step 128, a preliminary 3D model is instantiated using the output ofthe information fusions step. The model is named and all the appropriatedata structures are updated. The model is also textured at this step.This is done by setting up a constrained boundary value problem (BVP)with constrains defined by the feature point correspondence and usingtexture from the image(s) provided by the user. In an exemplaryembodiment, this is done using a technique similar to that presented in[45] for the face. The feature point correspondence between points onthe 3D model and those in the images is obtained using the segmentationresults from step 146. Alternatively, this correspondence data may beobtained through a user interface. An exemplary embodiment of such auser interface is discussed in reference to FIG. 11. A texture map forthe face is obtained by unwrapping a texture map from the input videosequence or input images using a technique similar to the texturemapping technique described in [46]. Before unwrapping the texture, theimages may be processed to complete missing or occluded regions (such asocclusion by hair, glasses, etc.) using shape space priors and symmetry.Skin tone is also identified at this step. In an exemplary embodiment,regions representing skin can be identified by converting the image to arepresentation in the HSV (Hue, Saturation, Value) color space or RGB(Red, Green, Blue) color space. Skin pixels have characteristic HSV andRGB values. By setting the appropriate thresholds for the HSV or RGBparameters, the skin regions may be identified. The skin reflectancemodel may incorporate diffuse and specular components to better identifythe skin. The variation of the pixel values (and higher orderstatistics) for example in RGB space can be used to estimate the skintexture. This texture is then used to fill in skin surfaces withunspecified texture values, for example, ears that are hidden behindhair. In an exemplary embodiment, skin texture is extracted from theface and used wherever necessary on the head and the body since the faceof a user is usually visible in the image or video. Similarly, textureis computed and mapped for teeth, hair, and the iris and pupil of theeyes. If image or video data is unavailable, a generic texture is used.The choice of a generic texture is based on other information providedby the user as part of other data 118 (eg. age, race, gender, etc.), ifavailable.

The model is then optimized at step 129. Optimization involves improvingthe model to better match the user. Optimization procedures similar tothose employed at step 125 and 153 are used at a global scale, ifnecessary or possible, again depending on user data and the actionfactors. Consistency checks are also made to ensure that scale andorientation of the different regions of the model are plausible andappropriate corrections are made if necessary. Textures on the model arealso optimized at this step if the action factors allow. This involvesoptimizations such as reilluminating the model so that the illuminationis globally consistent and so that the model can be placed in newillumination contexts. This is done in an exemplary embodiment usingtechniques similar to those used in [19, 20, 47]. Forward and backwardprojection (from the 3D model to the 2D image and vice-versa) may beapplied in a stochastic fashion to ensure consistency with the 2D inputimage, if provided, and to make finer modifications to the model, ifnecessary depending on action factors. The comparison of the projected3D model and the 2D image may be done in one or more feature space(s),for example in edge space. All of the actions performed are takendepending on the action factors as described earlier.

The method 110 then proceeds to step 130 at which the model is detailed.The photorealism of the model is enhanced and any special effects thatare required for NPR are added based on the action factors. Thephotorealism is enhanced, for example, by using bump maps for, say,wrinkles and incorporating subsurface scattering for skin. Facial hair,facial accessories and finer detail are also added to the model.

Method 110 then proceeds to the user modification step 131 at which theuser is allowed to make changes to the model if desired. These changesinclude, in an exemplary embodiment, changes to the skin tone,proportions of various body parts, textures (for example, the user mayadd scars, birthmarks, henna, etc.), etc. An easy to use user interfaceallows the user to make such changes as described later in thisdocument. Users are also allowed to set default preferences for theirmodel at this point. For instance, they may choose to have aphotorealistic model or a nonphotorealistic (NPR) model as their defaultmodel (NPR models may be multi-dimensional-1-D, 2-D, 2.5D, 3-D, 4-D orhigher). Users can also create several versions of their NPR model basedon their specific taste. Such NPR models can be constructed by simplyapplying a new texture or using algorithms such as those described in[48-50]. At any point during model construction, the method may ask theuser for assistance. The user is allowed to make changes to the model atany time. As the user ages, loses or gains weight, or goes throughmaternity, the model can be updated accordingly. As newer versions ofthe software are released, newer, more accurate versions of the modelmay be created using the information already supplied by the user orprompting the user to provide more (optional) information. All themodels created by the user are stored and the user is allowed to use anyor all of them at any time. The models created by the user are stored inthe user database 80 and are also cached on the client side 14 and 16for performance purposes.

The model generated before user modifications as well as the usermodifications and user data 111 are passed onto the learning step 132,the output of which is used to update the prior knowledge 112 in orderto improve the model construction method 110 over time. This can be doneusing reinforcement learning and supervised learning techniques such asGaussian process regression. In an exemplary embodiment, the manifoldsand the classifier used in the model construction process are updated.In an exemplary embodiment, if a model that is created is significantlyfurther away in distance from the existing exemplars of the classifierand has been found frequently, it is added as a new exemplar. At theconclusion of the user modifications step 131, a user model is created.

If the user provides more data 111, the method accesses the quality ofthe data, for example, the resolution of the images, the poly count ofthe meshes, etc. in order to determine if the newer data can improve themodel. If it is determined that the new data can improve the module, themethod 110 processes the data to improve the quality of the user modeland a new version of the model is created and stored. The measurementsof various body parts can be updated at any time as the user ages,gains/loses weight, goes through maternity etc.

The method 110 described above can be used for building models of otherobjects. For example, 3D objects for use in the virtual world. In anexemplary embodiment, the user can identify the class of the object(such as a pen, a laptop, etc.) for which a model is being created. Theclass of the object for which a model is being created is useful forselecting the appropriate priors for model construction for the givenobject from the prior knowledge 112. In an alternative embodiment, theclass of the object being considered can be automatically determined asdiscussed with reference to FIG. 49Q.

In an exemplary embodiment, a generative model for motion is used. Forexample, for the case of walking. users are allowed to tune variousparameters corresponding to a walking style such as a masculine/femininewalking style, a heavy/light person walking style, a happy/sad walkingstyle etc. Such generative models are learnt, in an exemplaryembodiment, using Gaussian process models with style and contentseparation using a technique similar to that used in [51].

When the action factors are very limiting, for example, on limitedplatforms such as a cell phone or a limited web browser, severalapproximations may be used to display a 3D model. In an exemplaryembodiment, on rotating a user model, the user is presented with a 3Dmodel of the user from a quantized set of views i.e. if a user rotateshis/her viewpoint, the viewpoint nearest to this user selected viewpointfrom a set of allowed viewpoints is chosen and displayed to the user. Inthis way, an entire 3D scene can be represented using as only as manyviewpoints as the system permits, thereby allowing a more compact andresponsive user experience. In an exemplary embodiment, if a genericuser model is used, precomputed views of the model corresponding todifferent viewpoints are used. In an exemplary embodiment, the apparelon a generic user model of a given size and the corresponding fit infois precomputed for various parameters (for example, for differentapparel sizes) and the appropriate view is displayed to the user. In anexemplary embodiment, the view may be an image or an animation such asone showing the user walking in a dress. As an exemplary embodiment ofhow a 3D environment can be displayed when the action factors arelimiting, static backgrounds may be used instead of dynamic one.Moreover, instead of displaying a fully 3D environment, a quantizedversion of the environment may be displayed i.e. as with the case of theuser model, when the user chooses to navigate to a certain viewpoint,the closest available viewpoint from a set of allowed viewpoints for theenvironment is chosen and displayed to the user.

Users can also choose to create a strictly 2D user model and try outapparel in 2D. This is one of the several options available for NPRmodels. In an exemplary embodiment, this is done by invoking the genericmodule 126 with a 2D option for the classifier i.e. the output of theclassifier is a 2D rigged mesh. The 2D classifier is built using thesame technique as described for the 3D models but using 2D rigged modelsinstead. Users can also draw a model of themselves. This can then beeither manually rigged through a user-interface or automatically using a2D form of the technique used in [31], in an exemplary embodiment. Usersalso have the option of creating their own 3D models, and using them fortrying out apparel and for various entertainment purposes such asplaying games and creating music videos containing their user model.

All data provided by the users and models constructed are saved in arepository. In an exemplary embodiment, an application programminginterface (API) may be available for developers to build applicationsusing this data. In an exemplary embodiment, an application could usethis data to determine items that fit a user as a user browses acatalog, as described later. In another exemplary embodiment, a mobiledevice or cell phone application could allow users to scan a bar code oran RFID (radio frequency identification) tag on an apparel in a realstore and see if the apparel fits the user. (Such scanning of bar codesor RFIDs and looking up of repositories can have other applications suchas scanning a food item to check if it is consumable by the user i.e.its ingredients satisfy the dietary restrictions of a user).

Reference is now made to FIGS. 7A-D which illustrate protocols forcollaborative interaction in exemplary embodiments. These protocols canbe used for a number of applications. These protocols are described nextfor the modes of operation of a Shopping Trip™. Other applications basedon these protocols are described later in this document. A user mayinitiate a shopping trip at any time. There are four modes of operationof a shopping trip: regular, asynchronous, synchronous and common. Inthe regular mode, a user can shop for products in the standardway—browse catalogues, select items for review and purchase desireditems. Whereas the regular mode of shopping involves a single user, theasynchronous, synchronous and common modes are different options forcollaborative shopping available to users. In the asynchronous mode, theuser can collaborate with other shoppers in an asynchronous fashion. Theasynchronous mode does not require that other shoppers the user wishesto collaboratively shop with, be online. The user can share images,videos, reviews and other links (of products and stores for instance)they wish to show other users (by dragging and dropping content into ashare folder in an exemplary embodiment). They can send them offlinemessages, and itemized lists of products sorted according to ratings,price or some other criteria. Any share or communication or otherelectronic collaborative operation can be performed without requiringother collaborators to be online, in the asynchronous mode at the timeof browsing. The synchronous and common modes require all collaboratingmembers to be online and permit synchronized share, communication andother electronic collaborative operations. In these modes, the users canchat and exchange messages synchronously in real-time. In thesynchronous mode, ‘synchronized content sharing’ occurs. Reference ismade to FIG. 20 to describe this operation in an exemplary embodiment.Users involved in synchronized collaboration can browse products andstores on their own. ‘Synchronized content sharing’ permits the user todisplay the products/store view and other content being explored byother users who are part of the shopping trip by selecting the specificuser whose browsing content is desired, from a list 244 as shown in FIG.20. For example, consider a shopping trip session involving twousers—user 1 and user 2, browsing from their respective computingdevices and browsers. Suppose user 1 and user 2 are browsing products byselecting “My view” from 244. Suppose user 1 now selects user 2 from theview list 244. As the selected user (user 2) browses throughproducts/stores, the same content is displayed on user 1's displayscreen thereby synchronizing the content on the display screens of users1 and 2. User 1 may switch back to her view whenever she wants andcontinue browsing on her own. Similarly, user 2 can view the content ofuser 1 by selecting user 1 from the switch view list. In the commonmode, users involved in the collaborative shopping trip aresimultaneously engaged in browsing products or stores on their displayscreens. This mode can assume two forms. In the first form, a user isappointed as the ‘head’ from among the members of the same shoppingtrip. This head navigates/browses products and stores on their displayscreen and the same view is broadcast and displayed on the screens ofall users of the same shopping trip. In the second form, all users cannavigate/browse through product, store or other catalogues and virtualenvironments and the information/content is delivered in the sequencethat it is requested (to resolve user conflicts) and the same content isdisplayed on all user screens simultaneously using the protocol that isdescribed below. In the common mode, all the users are engaged in ashopping trip in a common environment. This environment may be browsedindependently by different members of the shopping trip leading todifferent views of the same environment. The system in FIG. 20 involvingsynchronous collaboration between users may be integrated with a ‘OneSwitch View’ (OSV) button that allows users to switch between user viewsjust by pressing one button/switch, which may be a hardware button or asoftware icon/button. The user whose view is displayed on pressing theswitch is the one on the list following the user whose view is currentlybeing displayed, in exemplary embodiment. This OSV button may beintegrated with any of the collaborative environments discussed in thisdocument.

The techniques for accomplishing each of the four modes of operation aredescribed next in an exemplary embodiment. Reference is now made to FIG.7A where the regular mode of operation of a shopping trip is shown. Aninstance of a client 201 in the regular mode of operation makes arequest to the server application 22 to view a product or a store orother data. In exemplary embodiment, the request can be made using HTTPrequest, RMI (remote method invocation), RPC (remote procedure call).The client instance then receives a response from the server. Referenceis now made to FIG. 7B where an asynchronous mode of operation is shownin an exemplary embodiment. In this case, the user instance 201 makes arequest to the server. A list 203 of shopping trip members and theirinformation is maintained on the server for any given user. The list 203is a list of users that have been selected by the client C6111 toparticipate in the shopping trip. In response to the client's request,the server then sends a response to the client 201 with the requestedcontent. If the item is tagged for sharing, the server adds it to a listof shared items for that user. Other users on the shopping trip mayrequest to view the shared items upon which the server sends therequisite response to this request. For instance, a user may view aproduct while browsing and may tag it as shared or add it to a sharebin/folder. For instance, a user (C6111) may view a product and add itto a share bin. Other users (C6742, C5353) may then view the items inthat bin. The shopping trip members list 203 may also be stored locallyon the client's side in an alternative exemplary embodiment. Referenceis now made to FIG. 7C where the synchronous mode of shopping is shownin exemplary embodiment. When a client instance 201 makes a request tothe server to view a product, for example, an appropriate response issent not only to the client requesting the information but also to allmembers on the shopping trip list who have selected that client'sbrowsing contents (refer FIG. 20). In another exemplary embodiment, thesynchronous mode works as follows: (1) A user, say USER1, visits aproduct page. (2) The product is registered in a database as USER1'slast viewed page. (3) If another user, say USER2, has selected theoption to show USER1's view, their view is updated with USER1's lastviewed product. (4) When USER2 selects USER1's view, the view is updatedevery 3 seconds. (If there is no activity on part of USER2 for a givenperiod of time, USER2's client application may pause polling thedatabase to save bandwidth and other computational resources. Uponreactivation by USER2, view updating may resume). Thus, updating of theviews may be server driven or client driven. Users can specify useraccess privileges to content that belongs to them. For example, they canset access privileges to various apparel items in their wardrobeallowing other users to access certain items and denying access tocertain others. An icon notifies the user if the current view is beingbroadcast. The history of a trip is also available to the users. In anexemplary embodiment, this is done by showing the user the items thatwere registered in the database in step (2) above. This history can alsobe downloaded and saved by the users and can be viewed later. Referenceis now made to FIG. 7D where the common mode of a shopping trip is shownin exemplary embodiment. In this figure, it is shown that severalclients can simultaneously make a request and simultaneously receive aresponse. At any given time, any of the clients can send a request tothe server to view an item, to explore an item (as discussed inreference to FIG. 36), etc. in exemplary embodiment. The following is adescription of the communication protocol for the common mode ofoperation of a shopping trip. When a client sends a request to theserver, it also monitors a channel on the server (could be a bit or abyte or any other data segment on the server in exemplary embodiment) tosee if there any simultaneous requests made by other users. If nosimultaneous requests are detected, the client completes the request andthe server responds to all clients in the shopping trip with theappropriate information requested. For instance, if a catalogue item isviewed by one of the users, all other clients see that item. As anotherexample, if a client turns over a 3D item, then all other clients seethe item turned over from their respective views. If however, asimultaneous request is detected at the channel, then the client abortsits request and waits for a random amount of time before sending therequest again. The random wait time increases with the number ofunsuccessful attempts. If the response duration is lengthy, thenrequests are suspended until the response is completed by the server, inexemplary embodiment. Alternatively, a conflict management scheme may beimplemented wherein the client also monitors the server's response for apossible conflict and sends the request when there are no conflicts. Inyet another exemplary embodiment, the server may respond to requests ifthere are no conflicts and may simply pause if there is a conflict.These protocols also apply to peer-to-peer environments with the sourceof the data being the server and the requesting party being the client.

While viewing products, the content from audio and video channels ofusers on the shopping trip, and also the output of common(collaborative) applications (such as a whiteboard-like overlay thatusers can use to mark items on the web page or in the environment, orwrite and draw on) can also be shared simultaneously. In an exemplaryembodiment, for the asynchronous mode, the user may tag an item forsharing and add it to a bin along with a video, audio and/or textmessage. When other users request to see items in this bin, they areshown the product along with the audio, video or text message. Inexemplary embodiment, for the synchronous mode, the audio channels forall the users are added up and the video channel for whichever user'sview is selected (FIG. 20) is shown. For the common mode of operation,in an exemplary embodiment, the audio channels from the users on theshopping trip are added up and presented to all the users while thevideo stream may correspond to the user who has just completed sending arequest successfully through the common mode communication protocoldescribed above. Sessions may be saved as described before. The viewsand the timeline during any session can be annotated. These pieces ofinformation are cross-referenced to enable the user to browse by any ofthe pieces of information and view the corresponding information.

For each of the above modes, the clients may also interact in a peer topeer fashion as opposed to going through a server. In an exemplaryembodiment, in the synchronized mode, if the user makes a request for awebpage to the server, then that information can be passed on to theother clients on the shopping trip via a peer to peer protocol. A usermay also be engaged in multiple shopping trips (in multiple shoppingtrip modes) with different sets of users. Additionally, sub-groupswithin a shopping may interact separately from the rest of the groupand/or disjoin the rest of the members of the shopping trip and thenlater resume activities with the group.

While operating in any of these modes, the user has the option to turnon an ‘automatic’ mode feature whereby the system engages the user in aguided shopping experience. In an exemplary embodiment, the user mayselect items or categories of items that the user is interested in andspecify product criteria, preferences and other parameters. The user mayalso specify stores that the user is interested in browsing. Once thisis done, the system walks the user through relevant products and storesautomatically for a simulated guided shopping experience. The automatedmode may be guided by a virtual character or a simulated effigy or areal person. The user can indicate at any time if she wishes to switchto the manual mode of shopping. The modes of operation presented herefor shopping can be applied to other collaborative applications. Forinstance, going on a field trip, or virtual treasure hunt, or sharingapplications as discussed with reference to FIG. 49 O.

Reference is now made to figures that describe the system 10 in greaterdetail, through sample images that are taken from the system 10. Thesample images describe the operation of the system 10 with examples thatare provided through sample screen shots of the use of the system 10.

Reference is now made to FIG. 8 and FIG. 31, where a sample main pagescreen 250 is shown, in an exemplary embodiment. The sample main screen250 is used for purposes of example. The main screen 250, in anexemplary embodiment presents the user with various options. The optionsin an exemplary embodiment include the menu options 252. The optionsmenu 252 allows a user to select from the various options associatedwith the system 10 that are available to them. In an exemplaryembodiment, the options menu allows a user to select tabs where they canspecify further options related to their respective environment 620,friends 622 and wardrobe 624 as has been described in FIG. 5. Users cansearch the site for appropriate content and for shopping items using thesearch bar 632; they can browse for items and add them to their shoppingtrolley 628 which dynamically updates as items are added and removedfrom it; and complete purchase transactions on the checkout page 626.The options that have been provided here, have been provided forpurposes of example, and other options may be provided to the user uponthe main page screen 250. Furthermore, users can choose and set thetheme, layout, look and feel, colours, and other design and functionalelements of the main and other pages associated with their account onsystem 10, in the preferences section 630. In an exemplary embodiment,users can choose the colour scheme associated with the menu options 252and the background of the main and other pages. The local applicationdescribed further below is launched on clicking the button 254. Thestatus bar 256 displays the command dressbot: start which appears as thelocal application is started. Button 258 starts the model creationprocess. When the local application 271 is running on the local machine,a notification 634 is displayed inside the browser window 250. Alongwith apparel shopping and modeling, users can engage, with their virtualmodel and with other users, in collaborative activities which include,in exemplary an embodiment, participating in virtual tours and visitingvirtual destinations 636; taking part in virtual events 638 such asfashion shows, conferences and meetings etc, all or some of which maysupport elements of augmented reality. A media player or radio may beavailable/linked available in the browser in an exemplary embodiment640, Featured apparel items 642 and other current offers or news orevents may also appear on the main page 250 in an exemplary embodiment.

Reference is now made to FIGS. 9 to 13, to better illustrate the processby which a 3D user model is created. As described above, the 3-D usermodel is created by first receiving user input, where the user suppliesrespective images of themselves as requested by the system 10. Referenceis now made to FIG. 9, where a sample image upload window is shown in anexemplary embodiment. The image upload window is accessible to the userthrough accessing the system 10. As described above, in an exemplaryembodiment, the system 10 is accessed through the Internet. The sampleupload window 260 is used to upload images of the user that are thenused by the system 10 to generate the user model. As shown in FIG. 9,the user is requested to upload various images of themselves. The userin an exemplary embodiment uploads images of the facial profile, sideperspective and a front perspective. In an exemplary embodiment, theuser is able to upload the images from their respective computing deviceor other storage media that may be accessed from their respectivedevice.

Reference is now made to FIG. 10, where a sample image of a clientapplication window 270 is shown. In an exemplary embodiment, the clientapplication 16 resident, or associated with the computing device causesa client application window 270 to be displayed to the user when theuser model is being created. The client application can request andsubmit data back to the server. The protocol for communication betweenthe application 16 and server 20 is the HTTP protocol in an exemplaryembodiment. The application 16, in an exemplary embodiment initiatesauthenticated post requests to a PHP script that resides on the portalserver and that script relays the requested information back to theapplication 16 from the server 20. People are comfortable with shoppingon the internet using a browser and with monetary transactions through abrowser. In order to provide the user with a rich experience, a rich 2Dand/or 3D environment is desired. Such an environment can be acomputational burden on the portal server. To reduce the computationalload on the portal server, the computationally intensive renderingaspects have been pushed to the client side as an example. In anexemplary embodiment, this computational efficiency can be achievedthrough the use of a local stand-alone application or a browser plug-in,or run within a browser, or a local application that interacts with thebrowser and portal server 20. The current implementation, in anexemplary embodiment, involves a local application 271 that interactswith the browser and the portal server and is a component of the clientapplication 270. In a typical setting, the local application and thebrowser interact with each other and also with the portal server 20,which in turn interacts with other components of the internet. Each ofthe modules of the portal server 20 may have a corresponding module onthe client application. This may be a part of the browser or localapplication 271, the browser or a combination of the two. The browserand the local application interact in an exemplary embodiment, viaprotocols like HTTP and this communication may take place via the portalserver 20 or directly. The purpose of the local application 271 is toenable computationally intensive tasks to be carried out locally such ascomputations required for 3D renderings of the apparel, the user's modeland the environments. This gives the appearance of running 3D graphicsin a browser. This permits online transactions within the browser(buying apparel) and at the same time gives the user a rich experienceby using the power of the local machine and not overburdening theserver. For those users who are not comfortable with downloading thelocal application 271, a 2D, 2.5D or less sophisticated 3D rendering ofthe graphics is displayed within the browser. Details of thebrowser-local application interaction are described next. In anexemplary embodiment, on a Windows® platform, registering the protocolassociates a keyword with the local application 271 on the user's systemin the registry. Thus, when the start application button 254 is pressed,the local application 271 is launched. When a user clicks on the ‘tryon’ button from the fitting room or wardrobe, a notification is sent tothe local application indicating that the user wants to try an apparelitem. A callback function is implemented within the local applicationthat listens for such notifications. When a notification is received,the appropriate callback function is invoked. This callback functionthen queries the portal server or browser for the appropriate parametersand renders the scene. For example, clicking on an apparel item in thefitting room prompts the browser to send the command “dressbot:tryon=5”to the local application which then places the item with ID=5 on theuser model. The gathering of information from the server is done usingHTTP. Such a framework leverages the advantages of both familiarexperience of a browser and the computational power of a localapplication. The above procedure and details have been described as anexemplary embodiment and may be implemented with other techniques. In analternative embodiment, local application features may be implemented aspart of a web browser.

By accessing the user model creation functionalities on the user's localcomputing device, the speed at which the model is generated and thenmodified (through the user's commands) is increased. The applicationwindow 270 displays to the user the current state of the model, andallows the user to perform various modifications to the user model, asdetailed below.

As described above, the user is able to modify the respectivemeasurements that are associated with a preliminary user model that hasbeen generated. The measurements specified by the user may be specificmeasurements that more closely resemble the user's physical profile.However, the measurements that are specified may also be prospectivemeasurements, where the user may wish to specify other measurements. Forexample, the user may specify measurements that are larger than theircurrent measurements, if for example, they wish to model maternityclothes. Also, the user may specify measurements that are smaller thantheir current measurements, thereby providing prospective looks withregards to what a user may look like if they were to lose weight.

The head and face region of the user's model is simulated by themodeling module 50 utilizing images of the user's face taken fromdifferent angles. The face generation process may be completelyautomated so that the modeling module 50 synthesizes the model's face byextracting the appropriate content from the user's images without anyadditional input from the user or it may be semi-automated requiringadditional user input for the model face generation process. Referenceis now made to FIG. 11, where a sample facial synthesis display window280 is shown illustrating a semi-automated facial synthesis procedure.The reference image 282 shows the user where to apply markers on theface i.e., points on the face to highlight. The sample image 284, in anexemplary embodiment shows points highlighting regions of the user'sface corresponding to the markers in the reference image 282. Themodeling module 50 may require additional inputs from the user tofurther assist the face generation process. This input may includeinformation on facial configuration such as the shape or type of faceand/or facial features; subjective and/or objective input on facialfeature dimensions and relative positions and other information. Thetype of input acquired by the modeling module 50 may be in the form oftext, speech or visual input. Additionally, the modeling module 50 mayprovide options to the user in order to specify various areas/pointsupon the respective area of the model that they wish to make furthermodifications/refinements/improvements to. It may then be possible totweak or adjust certain facial features using adjustment controls as inthe case of the slider control feature for tweaking body measurementsdescribed later in exemplary embodiment. To be able to better illustratethe how the user may make modifications to the user model in anexemplary embodiment, reference is made now to FIGS. 12 to 13. Referenceis now made to FIG. 12A, where a sample measurement window 290 is shown,in an exemplary embodiment. The measurement window 290 allows the userto specify empirical data that is used to generate or modify the usermodel. The user is able to specify the measurements through aid of agraphical representation that displays to the user the area or regionfor which a measurement is being requested. In addition. videos and/oraudio may be used to assist the user in making measurements. When a userdoes not specify the measurements that are to be used, default valuesare used based on data that is computed from the respective images thatthe user has provided. Measurements associated with a user's waist havebeen shown here for purposes of example as the user may specifymeasurements associated with other areas of their body as describedabove. The user may specify various modifications of the user model thatare not limited to body size measurements. Such modifications mayinclude, but are not limited to, apparel size, body size, muscle/fatcontent, facial hair, hair style, hair colours, curliness of hair, eyeshape, eye color, eyebrow shape, eyebrow color, facial texturesincluding wrinkles and skin tone.

Reference is now made to FIGS. 12B and 12C, where a sample image of aconstructed model image 300 and 302 are shown, respectively. The modelimage window allows the user to inspect the created user model, byanalyzing various views of the created model. Various features areprovided to the user to allow the user to interact with the createdmodel, and to be able to better view various profiles associated withthe model. Features 303, 304, 305 and 306 are depicted as examples.Pressing button 306 presents the user with options to animate the usermodel or the environment. In an exemplary embodiment, the user may bepresented with animation options on the same page or directed to adifferent page. The user may be presented with specific presetexpressions/actions in a menu, for example, to apply on their usermodel. In an alternate exemplary embodiment, the user may animate theirmodel through text/speech commands or commands expressed via othermeans. The user may also choose to synchronize their model to their ownexpressions/actions which are captured via a video capture device suchas a webcam for example. The user is also provided with environments toembed the character in as it is animated. Icon 306 allows the user tocapture images of the model, or to record video sequences of modelanimation, which may then be shared by the user with other users. Thefacial icon 303 when engaged causes the face of the generated model tobe zoomed in on. The body icon 304 when engaged causes the entire usermodel to be displayed on the screen.

Reference is now made to FIG. 13A, where a set of sample nonphotorealistic renderings are shown. Specifically, exemplary embodimentsof non photorealistic renderings 310A, 310B, and 310C are shown. The nonphotorealistic renderings display a series of images, illustratingvarious views that may be seen of a user model. The respectivenon-photorealistic renderings illustrate the various rotations of theuser model that the user may view and interact with. Further, nonphotorealistic renderings 310A and 310B illustrate how the user maymodify the wrist dimensions of the model. In an exemplary embodiment,the user may select areas on the user model where they wish to modify arespective dimension. For example, by engaging the user's model atpre-selected areas or ‘hotspot’ regions, a window will be displayed tothe user where they may specify alternative dimensions. FIG. 13A showsthe wrist being localized via a highlighted coloured (hotspot) region312 as an example. The dialog box 313 containing slider controls can beused by the user to adjust measurements of the selected body part and isshown as an exemplary embodiment. FIG. 13B shows more sample images ofhow users can make body modifications directly on the user model usinghotspot regions 312.

Reference is now made to FIG. 13C which shows a sample ruler for takingmeasurements of the user model which may be displayed by clinking on aruler display icon 316. This ruler allows the user to take physicalmeasurements of the user model and to quickly check measurementsvisually. The ruler may also prove useful to the user in cases wherethey wish to check how a given apparel or product affects originalmeasurements. In an exemplary embodiment, the user may try on differentpairs of shoes on the user model and check how much the height changesin each case.

Reference is now made to FIG. 14, where a sample environment managerwindow 330 is shown in an exemplary embodiment. The environment moduleas described above, allows a user to choose respective environmentbackgrounds. The system 10 has default backgrounds that that the usermay select from. Also, the user is provided with functionality thatallows them to add a new environment. By uploading an image andproviding it with a name, the user is able to add an environment fromthe list that they may select from. Various types of environments may beadded, including static environments, panoramic environments,multidimensional environments and 3-D environments. A 3D environment canbe constructed from image(s) using techniques similar to those presentedin [44].

Reference is now made to FIG. 15A, where a sample user model environmentimage 340 is shown containing a photorealistic user model. The image 340is shown for purposes of example, and as explained, various backgroundenvironments may be used. Further, the user model that is shown in FIG.15A, has been customized in a variety of areas. Along with the apparelthat the user has selected for their respective user model, the user isable to perform different customizations of the model and environment.Examples of which are shown here for purposes of example. With referenceto labels 342, the user has customized the hair of the user. Thecustomization of a user model's hair may include, the style, hair andcolour. With reference to label 344, the environment may be customized,including the waves that are shown in the respective beach environmentthat is illustrated herein. With reference to label 346, one example ofthe types of accessories that the user can adorn their respective modelwith are shown. In this example image, a bracelet has been placed uponthe user model's wrist. As a further example of the various accessoriesthat may adorn the model, reference is made to label 348, wherein shoesare shown upon the respective user model. Reference is now made to FIG.15B where some aspects of collaborative shopping are illustrated. Usermodel views may be shared between users. Users may also interact viatheir model in a shared environment. In an exemplary embodiment, window354 shows two user models in a shared window between users. Productcatalogue views 355 may also be shared between users. For example, viewsof mannequins displaying apparel in product display window 355 may beshared with other users using the share menu 358. In another exemplaryembodiment of a collaborative shopping feature, views of shopping malls356 may be shared with other users as the user is browsing a virtualmall or store.

Reference is now made to FIG. 32 and FIG. 33, where more sampleenvironments and the types of activities the user can engage in withtheir virtual models are shown in exemplary embodiment. FIG. 32 depictsan environment where a fashion show is taking place and where one ormore users can participate with their virtual models 650. Theenvironment settings, theme and its components 652 can be changed andcustomized by the user. This is a feature that designers, professionalor amateur, and other representatives of the fashion industry can takeadvantage of to showcase their products and lines. They may also be ableto rent/lease/buy rights to use the virtual model of users whom theywould like to model their products. Users may also be able topurchase/obtain tickets and attend live virtual fashion shows withdigital models featuring digital apparel whose real and digital versionscould be bought by users. FIG. 33 shows a living room scene which can befurnished by the user with furniture 654 and other components from anelectronic catalogue in an exemplary embodiment. Users may use theirmodel 650 to pose or perform other activities to examine the look andfeel of the room, the setting and furnishing, which they may replicatein their own real rooms. This feature is further representative of‘interactive’ catalogues where users are not just limited to examiningdifferent views of a product before purchasing it from an electroniccatalogue but are able to examine it in a setting of their choice,interact with it via their virtual model or directly, acquire differentperspectives of the product in 3D, and get acquainted with enhanceddepictions of the look and feel of the product. Environments will alsobe available to users that change with time or other properties. Forinstance, an environment that represents the time of day may changeaccordingly and show a daytime scene (with the sun possibly and otherdaytime environment components) during daylight hours which changes torepresent the way the light changes and dims during the evening timewhich subsequently changes into a night scene with the appropriatelighting, other environmental conditions and components in an exemplaryembodiment. Environments that reflect the weather would also beavailable. Retailers would have the opportunity to make available theirapparel digitally with the appropriate environments. For instance,galoshes, raincoats, umbrellas and water-resistant watches and jewelrymay be featured in a rainy scene. Users may also customize/programscenes to change after a certain period of time, in an exemplaryembodiment. For instance, they can program a given scene or scenecomponents to change after a fixed period of time. User models may alsobe programmed to reflect changes over time such as ageing, weightloss/gain etc.

Reference is now made to FIG. 34, where a sample virtual model is shownin a customized music video that the user has generated. This figure isshown in exemplary embodiment and it illustrates the differentactivities the user can engage their virtual model in; the differentenvironments they can choose to put their model in as well as theexpression/action animation control they have over their virtualcharacter model. Display window 672 shows the virtual model singing in arecording studio; display window 674 shows the model driving in a sportscar while display window 676 shows the model waving and smiling. Theuser can choose to combine the different scenes/animations/frames toform a music video as depicted in FIG. 34. Another feature is avoice/text/image/video to song/music video conversion. Users can uploadaudio/video/text to the system and the system generates a song or amusic video of the genre that the user selects. As an example, a usercan enter text and specify a song style such as ‘country’ or ‘rock’ andother styles. Based on this, the system generates a voice that sings thewritten text in the specified style. The voice may also be selected(based on samples provided by the system) by the user or picked by thecomputer. (Given some content, the system can find related words to makerhymes while adhering to the provided content. In an exemplaryembodiment, this can done by analyzing phonemes and looking up in athesaurus to find rhyming words where necessary). For purposes ofincreasing computational efficiency, the system 10 may provide the userwith pre-rendered scenes/environments where the music and environmentcannot be manipulated to a great degree by the user but where renderingof the character model can occur so that it can be inserted into thescene, its expressions/actions can be manipulated and it can be viewedfrom different camera angles/viewpoints within the environment. Userscan save and/or share with other users the various manifestations oftheir user model after manipulating/modifying it and the animation/videosequence containing the model in various file formats. The modified usermodel or the animation/video sequence can then be exported to otherlocations including content sharing sites or displayed on the profile orother pages. In an exemplary embodiment, users may want to share theirvacation experiences with other users. In such a case, users can showtheir character model engaged in different activities (that they wereinvolved in during their vacation), against different backdropsrepresenting the places they visited. This could also serve as anadvertising avenue for the tourism industry. The model may be animatedto reflect the status of the user and then displayed on the profile pageto indicate other members of the status of the user. For instance, thecharacter model may reflect the mood of the user—happy, excited,curious, surprised etc. The model may be shown running(image/simulation/video) in a jogging suit to indicate that the user isout running or exercising, in one exemplary embodiment. The brand of thedigital apparel may appear on the apparel in which case featuring themodel on the profile page with the apparel on would serve as brandadvertisement for that apparel.

Along with the specification of accessories, the users as explainedbelow, are able to modify textures associated with the user model. Withreference to label 350, an example of the texture modification of a usermodel is illustrated. Skin color can be changed by changing HSV or RGBand skin texture parameters as discussed with reference to step 128 inFIG. 6A. Skin embellishments such as henna or natural skin pigmentationsuch as birthmarks etc. can be added by using an image of the respectiveobject and warping it onto the user model where placed by the user.Color palettes (a colour wheel for example) may be provided withdifferent variations of skin tones for users to pick a skin tone.Similar palettes may exist for makeup application.

As described above, the community module allows the respective user tointeract with other users of the system 10. Along with other users ofthe system 10, users are also able to invite other members to be usersof the system 10.

The system 10 allows for multiple methods of interaction between therespective users of the system. The various methods of interaction aredescribed herein. One such method of interaction is the concept of acollaborative shopping trip that is described in further detail herein.By having multiple users participate in a shopping trip, where users ofthe system 10 may interact with one another with respect to items ofapparel or other products, each other's models, messages, and picturesor images. By creating and participating in a shopping trip as describedherein, the real-world concept of inviting friends, shopping, andreceiving their respective feedback on purchased items is emulatedthrough the system 10.

Reference is now made to FIG. 16, where a sample image of a shoppingtrip management panel 360 is shown in an exemplary embodiment. Theshopping trip management panel 360 allows users to manage existingshopping trips that they have created, or to create new shopping trips.Once the user has created a new shopping trip, the user may then inviteother users to become members of their shopping trip as described withreference to FIG. 40. The user may send invites for shopping trips andother synchronized collaboration via the messaging service providedthrough system 10 and through other online or offline modes of messagingincluding email, SMS or text, chat and other means. Notifications canalso be sent to users on social networking sites inviting them forcollaborative activities. Users can also access past sessions that theywere on through the panel 360.

Reference is now made to FIG. 17, where a sample friends manager window370 is shown in an exemplary embodiment. The friends manager window 370allows users to invite other users to join them in their shopping trips.As illustrated with reference to FIGS. 17 and 18, the system 10 allowsfor friends that are associated with the system 10, and those that maybe associated with one or more other community networking sites to beinvited. Community networking sites include sites such as Facebook, orMy Space and others that allow their API to be used by externalapplications In an exemplary embodiment, a user's list of friends fromsocial networking sites may be displayed within the system 10. In anexemplary embodiment, a procedure for accessing friends on a user'sFacebook account is presented in FIGS. 39 to 42. FIG. 39A presents thesequence of events leading to the availability of one's Facebook friendson their account in system 10. FIGS. 39B to 39D display magnified viewsof each of the windows shown in FIG. 39A. Upon logging into system 10,the user can view his account information 716 as shown in FIGS. 39A and39B. A provision 719 exists on the account page 716 for signing intoFacebook, an external social networking site, which will facilitateaccess to Facebook account resources (other social networking sites maybe present and accessed through system 10). As illustrated in FIGS.39A-B, this will take the user to their login page 717 on Facebook, uponwhich the user may log in to his Facebook account 720. This will takethe user back to their account 718 on system 10, this time with accessto the user's Facebook friends 721 and other information availablethrough their account on system 10 as shown in FIGS. 39C and 39D. Whenthe user decides to logoff from their account on system 10, the user isasked if he/she wishes to logoff from Facebook as well. Users are alsoable to import data from external sites. For example, contactinformation or images may be imported from social networking sites suchas Facebook, Personal data such as measurements of the user's body maybe imported from a repository containing information on the user'smeasurements 115 described with reference to FIG. 6A. Pictures may beuploaded to the users account on system 10 from a photo sharing site.

Users are able to invite friends from the community network sites tointeract with. Upon requesting that a friend from a community networkingsite join in a shopping expedition, the friend when accessing theiraccount in the community network site, receives a notification that arequest has been made. The user may choose to accept or reject therequest.

Reference is now made to FIG. 18, where a sample system friendshipmanagement window 380 is shown in an exemplary embodiment. The systemfriendship manager is used to manage a user's relationship with otherusers of the system 10. The manager window 380 lists a user's friends,along with friend requests that are still pending. Search functionalityis also provided for, where a user may search for other users byentering their names

Reference is now made to FIG. 19, where a sample chat window 390 isshown in an exemplary embodiment. The chat window in an exemplaryembodiment may be created for every shopping trip that is associatedwith the user. Through the chat window 390, users are able to engage inan interactive chat session with one or more other users. The shoppingtrip feature allows two or more users to collaborate while shoppingonline. This may entail limited or full sharing of account resources forthe duration of the shopping trip. In an exemplary embodiment, users canview the contents of each other's shopping carts, shopping lists,wishlists, fitting rooms, user models, and share audio play lists andother resources. They can set and view shared ratings, feedback,comments and other user-specified information regarding a product. Theycan mark items with user tags that can be shared between members of theshopping trip. Additionally, users can shop in collaborativeenvironments wherein, in an exemplary embodiment, users can agree on aselected virtual mall environment and browse virtual stores and itemsconcurrently. Reference is now made to FIG. 20 where a collaborationinterface for a shopping trip 240 is shown in exemplary embodiment.Members of the shopping trip are shown by clicking on button 241. Here alist of stores that the users can browse is presented in panel 242. Thispanel may show all the stores subscribing to system 10. Alternately, themembers of the shopping trip may add stores of interest to them orremove stores from the panel. The store names may be presented as a listor on a map of a virtual or real mall in an exemplary embodiment. Inthis example, the stores appear in a list 242. Users can select theshopping environment 243 for a shopping trip session. The shoppingenvironments may be animated and/or video/image representations offictional malls or real malls, or other manifestations as describedpreviously with reference to the environment module 56, the shoppingmodule 60, and the entertainment module 66. The shopping environmentsmay incorporate a mode with augmented reality features, which weredescribed previously with reference to the shopping module 60. Users canengage in an interactive session within a store environment in 243, asin FIG. 46, when operating via this mode. Users can also view productcatalogues and individual products in 243. Users can also view stores in243 that are available on the retail server 24. Users can acquiredifferent product views, and examine products in 3D in 243.Additionally, a mode with physics based effects may be incorporated tosimulate product look and feel as well as simulate realistic interactionwith the product virtually via display 243. In an exemplary embodiment,information of a specific mall may be provided in the form of audio andvisual (video/image sequences and/or text) feeds via 243 when a userselects a particular mall. This way, users would be able to shopremotely in malls or stores located in other countries such as Paris,Milan, New York and other cities and shopping hubs. Individual stores inthe mall may also transmit live feeds via webcams, in exemplaryembodiment, (and/or other image, video capture devices) which users canview in 243. This feed content may incorporate information on the lateststock, new arrivals, promotions, sales, window displays, shelf contents,inventory, salespeople, store arrangements, live reviews and otherinformation relevant to the store. Miscellaneous information such as jobopenings in the store may also be included. Feed information would beuploaded via a web page onto the portal server 20. This informationwould be broadcast in 243 to clients requesting the feeds. Tools may beavailable to vendors to edit feed information. For instance, video feedinformation may be edited, image information may be enhanced throughphotorealistic effects etc. Feed information would provide a mode ofadvertising to stores. The facility to publish feed content may beavailable through an independent plug-in or software application tostores. The feed information does not necessarily have to be generatedfrom physical store locations. This information may be provided by thebrand or store head office. In the case that a customer browses a mall,an association file would assist in linking stores and/or brands tomalls in which they have physical presence. Feed content may behyperlinked. In exemplary embodiment, as customers browse store feeds,they may click on a product item to browse its details such as thosedescribed with reference to 22. Other details may be included such asinventory details of a particular item; product ratings (maybe assignedby customers or style consultants); style information; links to otherproducts that can be worn with it and/or other similar styles in thestore. The hyperlinks may be represented by icon such as animated tags.Other hyperlinks that may be present in the store feeds include links toelectronic fashion magazines or videos containing information or demosor reviews about specific store products, styles, brands, etc.

On a shopping trip that involves more than one user, shopping tripmembers may choose to shop collaboratively. There are several ways toengage in a collaborative shopping trip, as described previously in thisdocument. A user may browse the chosen environment and/or products, andat any given time, the video, animation or image sequence informationthat is displayed on the user's screen while the user is browsing theenvironment and products is considered the specific user's ‘view’. Userscan choose to display the views of all members, which will appear on asplit-window screen in an exemplary embodiment. Alternatively, they canchoose to display a specific member's view on their screen or return totheir own view. Members on a shopping trip can switch between views 244of individual members browsing the common environment or product 243.Furthermore, users can choose to browse different digital manifestations245 of the environment and/or product such as streaming video, imagesequences, virtual simulation, augmented reality, other media content orany combination thereof. In the asynchronous mode, users candrag-and-drop and/or add items and products that they wish to share withother users from display screen 243 to a sharing folder, the contents ofwhich can be viewed by the members of the shopping trip at any time.Users may view and examine their own account resources such as theirvirtual/digital model, wardrobe and fitting room contents, shoppingcart, wishlist, image and other features during the shopping trip. In anexemplary embodiment, the user may view his resources in the window 246,by selecting from the menu 247. Currently, the user model is displayedin 246. Users can share their account resources such as their profileimages, shopping cart contents, character model and fitting room contentwith other members of the shopping trip. Shared information by otherusers is viewable in display window 248. By selecting from the tabbedmenu 249, shown here in an exemplary embodiment, a user can view theparticular resource of the members of the shopping trip in 248. Userscan add their virtual models to the environment which can be viewed bythe members on the shopping trip who have the required access andpermissions. Users on a shopping trip will be able to communicate witheach other via multiple-way conferencing, chat (which may include textand/or speech communication; 3D visualization and/or augmented realityviewing and interaction). FIG. 20 shows a chat window 390 in anotherexemplary embodiment, within the shopping trip scenario. FIG. 20 couldalso be used in other scenarios as well such as choosing a restaurant tovisit for dining. A user and their friends can collaboratively viewinformation on restaurants in 243. Visual 3D menus may be available forviewing restaurant meal choices, for receiving feed information onspecials, promotions, reviews and other relevant restaurant information.Users would also be able to collaboratively order a meal for take-outand review restaurant menus and other information online in order todecide where they would like to go for dining.

Reference is now made to FIG. 40 where an exemplary embodiment of theprocess joining a shopping trip through a user interface is shown. In anexemplary embodiment, this process proceeds as follows: When a userclicks on a “Go Shopping” button, he/she is presented with a screen withthree columns—left, middle, right. The column on the left lists allexisting shopping trips that the user's friends are currently engagedin. The user can choose to join any of these shopping trips by clickingon a “join” button. The user also has the option of searching for ashopping trip of interest. When a key word is searched for the relatedshopping trips are presented in the left column. The keyword could bethe name of a shopping trip or an item of interest that is being shoppedfor, or an occasion, as examples. When the user clicks on the name of ashopping trip in the left column, the members of that shopping trip areshown in the middle column. The user can also invite other friends byclicking on the name of a friend from the right column and then clickingon the “invite” button. (The right column includes a list of all theuser's friends. These friends include friends on from our shopping site,social networking sites such as Facebook, or friends from the virtualoperating system/immersive system described in this document. The usercan also search for a name of friend to add to the shopping trip. If thefriend is found, the name appears in the right column and the user caninvite the friend by clicking on the invite button). The friend thenreceives an invitation via a notification on a social networking site, aphone call, an SMS, an email or other means as described before. Thefriend's name appears in the middle column in red until the friendaccepts the invitation. If the user's friend accepts the invitation,that friend's name appears in the middle column in blue. An orange colorindicates that the friend will be joining later. Other cues may also beused to display the status of the friend. The user can also initiate anew shopping trip by specifying a name and clicking on the “new” button.The user also has the option of removing friends from a shopping tripthat the user has initiated by clicking on the remove button under themiddle column. The user can start the shopping trip or resume a shoppingtrip by clicking on the “GO” button. The next screen presented onclicking “GO” is a screen listing cities, malls, and stores. The userscan pick any city, mall, or store to go to and shop via any of the modesof interaction of a shopping trip described earlier with reference toFIG. 7. At any given time, the user can be engaged in multiple shoppingtrips and can switch between any of the trips or add/remove friends bycoming back to this interface. The name of the shopping trip that theuser is currently viewing in appears on top as the user shops. Such aninterface is also used for going to events such as those described withrespect to the “hand and chill” feature (For example, as described withreference to FIG. 44). In an exemplary embodiment, the main shoppingpage includes two buttons—“Browse” and “Shopping Trip”. Clicking on“Browse” lets the user shop in the regular mode of shopping. Clicking on“Shopping Trip” loads the screen shown in FIG. 40.

Reference is now made to FIG. 41A-F where snapshots of a realization ofthe system discussed with reference to FIG. 20 are shown in an exemplaryembodiment. Upon visiting the site (in a browser in this case), the useris presented with the option of logging in or browsing in regular mode(as shown in FIG. 41A). After logging in, the user can click on the“Shopping Trip” icon from the top menu. As shown in FIG. 41B, thisbrings up the shopping trip screen discussed with reference to FIG. 40.Shown in the middle column are the friends that are on the selectedshopping trip. Friends that have not yet accepted the invitation to jointhe shopping trip are highlighted in red. Trip requests show up in thepanel on the right and/or as a Facebook notification and/or as an SMS,etc. depending on preferences specified by the user. A sliding chatwindow 390 can be used at any time. In an exemplary embodiment, shown inFIG. 41C is one instance of the synchronous mode of operation of ashopping trip in use. In an exemplary embodiment, after starting ashopping trip, users are presented with a list of stores that they cango to. On selecting a store to visit, the user is presented with a menu(menu on the left in FIG. 41C) for browsing through products. This menumay be customized for each store, for example, by providing the vendorswith an application programming interface (API) or by letting thevendors customize the menu and navigation options through the storeportal discussed with reference to FIG. 42. Item-dependent views arealso provided. Based on the content that is being viewed, an appropriateviewing method is used. For example, the method of displaying cosmeticsmay be different from that of displaying clothes. The chat windowenables the user to chat with a selected user (who could be on ourwebsite or on any other social networking site like Facebook or on achat application such as msn or via email or on a cell phonecommunicating via text such as through SMS or via voice by employingtext to speech conversion, in exemplary embodiments) or with all membersof a selected shopping trip. The panel on the right in FIG. 41C (but tothe left of the chat window 390) provides various options and controlsto the user as described earlier. The “My Friends Views” box in thepanel is similar to 244 described earlier. It enables the user to selecta view which could be the user's own view or any of the user's friend'sviews and interact with friends in the modes of operation discussed withreference to FIG. 7A-D, and described next in an exemplary embodiment.In the synchronous mode (which is the default mode), clicking on afriend's name in the “My Friends Views” displays the view 243 as seen bythat friend in the current user's view 243. In the common mode (whichcan be initiated by clicking on a ‘common’ icon next to the friend'sname), the view of the current user including navigation options becomesinteractable/controllable by all the friends who have been marked as‘common’. In the asynchronous mode, (which can be entered by clicking onthe “shared items” icon on the top menu as described below withreference to FIG. 41D), clicking on a friend's name lists items that arebeing shared asynchronously by that friend. The view 243 isundockable/dockable/movable/dragable to allow multiple viewssimultaneously and can also be minimized/maximized/resized. One way todo this is to drag out the view 243 which opens it in a new window thatcan be placed elsewhere. Multiple views may be opened at any given time.As shown in FIG. 41C in an exemplary embodiment, the multiple views areshown by numbers next to “My View”, or the user's friends' names in 244.This is particularly useful when viewing multiple items collaboratively.For example for mixing and matching; friends may find a skirt that theylike and may need to search for a top to go with it. An interfacesimilar to that described with reference to FIG. 45 can also be usedhere for mixing and matching. The panel is also undockable/dockable andcan be moved/dragged around and also be minimized/maximized/resizedbased on the users' preference. Under “My Friends Views”, users can alsosee which of the user's friends are online or are actively browsing.This is indicated by the color of a ‘person’ icon next to each name. Ashortcut is also located next to each of the friends' names to quicklyslide out the chat box 390 and chat with the friend. Users can alsoclick on a phone icon that lets the user talk to a friend or all membersof a shopping trip. In an exemplary embodiment this is done either overVoIP (Voice over Internet Protocol) or by dialing out via atelephone/cellular line through a modem. Users can also engage in avideo chat with their friends. Clicking on the radio on the left, bringsup options for the radio (such as a title to play, a playlist, volume,play individually, play the same music for all members of the shoppingtrip, etc.) in the view 243. These options can be set using the variousmodes of interaction as described above, Clicking on the “shared items”icon on the top menu brings the “My Shared Items” and “My Friends SharedItems” boxes in the panel as shown in FIG. 41D in an exemplaryembodiment. These boxes list the items that are posted by the user or bythe user's friends for sharing with others asynchronously. Clicking onthe “My Wardrobe” icon on the top menu brings up a “My Wardrobe” box inthe panel as shown in FIG. 41E in an exemplary embodiment. This boxlists the items that the user has in his/her wardrobe. In an exemplaryembodiment, items get added to the wardrobe once the corresponding realitems are purchased. Users can drag and drop items from the “MyWardrobe” box to the view 243 or can mark the items in “My Wardrobe” forsharing. Clicking on the “Consultant” icon brings up a “Chat with aconsultant” box in the panel as shown in FIG. 41F in an exemplaryembodiment. Users can add consultants from a list. Recommendations onstyle consultants by friends are also displayed. Users can share viewsand engage in an audio/video/text chat with consultants similar to theway they interact with their friends as described above. Consultants canalso participate in collaborative decision making through votesdescribed as described in this document. Upon clicking on the “CheckOut” icon, users are presented with the SPLIT-BILL screen as discussedwith reference to FIG. 21. Clicking on the “Logout” icon logs the userout of the system. The user's friends can see that the user has loggedout as the colour of the icon next to the name of the user under “MyFriends Views” changes. The user may join the shopping trip later andcontinue shopping. The user can exit from a shopping trip by clicking onthe shopping trip icon, which brings up the screen shown in FIG. 40 or41B, and then clicking on the “exit” icon next to the name of theshopping trip. The interface and system described here can also be usedto browse external websites and even purchase items.

Store feeds (which could be videos on the latest items in the store orthe items on sale in a store, or could also be streaming videos fromlive webcams in stores displaying items on sale) as described in thisdocument are also viewable in the screen 243. Users of the shopping tripcan not only access products offered by various stores but alsoservices. For example, a movie ticket purchase service is offered thatworks as follows in an exemplary embodiment: Suppose a bunch of friendswant to go out to watch a movie. These friends can go on our site. Onselecting the name of a cinema from a services menu, the users arepresented with a screen that displays the available locations for thecinema. Users can choose the location they want to go, or assign a headto decide on the location or let the system propose a location to go to.If they chose a location themselves, a majority vote is taken and thelocation corresponding to this majority is proposed as the location thatthey should go to. If all the users agree to go to the voted location,they can proceed to checkout/booking. Otherwise, the system proposesalternatives. If any of the users assigns a head, the choice of the headis taken as the choice of the user too. The system can also proposelocations. For example, it may calculate the location of a theater thatminimizes the travel for all the users on a shopping trip such as alocation that falls close to all the users. The system may also identifylocations where there is a special promotion or a sale or something todo in the proximity. It can make statements such as, “You can go to BlahTheater and then go for dinner at DinnerTime Restaurant which is onlyfive minutes away and food there is at half price today”. In anexemplary embodiment, this can be done by evaluating conditionalprobabilities that are constructed based on data from several users.After selecting the location, the users are presented with anotherscreen that lets them choose the movie that they would like to watch andthe show time. Trailers for each of the movies currently playing may beshown on this page and the users. The selection of movie titles and showtime proceeds in a similar manner to that of the location of a theater.Upon selection of a location, movie and time, the users proceed tocheckout at which point they have the option of using Split-Billfeatures if desired. (Users may simply state a movie they would like towatch and the system may propose the nearest location that plays themovie and that works with all the members of the shopping trip). Thismethod works with any of the modes of operation of the shopping trip. Inan exemplary embodiment, users can also watch the movie for whichtickets have been purchased online collaboratively. Further details arediscussed with reference to FIG. 44. Shopping trips can also work onmobile devices.

Users of the shopping trip can also collaboratively pick and choosedesigns, styles, colours, and other aspects of apparel, and share theiruser model or user data 111 to build customized apparel. Similarly,users can design a room and purchase furniture, or design, build and buyfurniture or other items. Collaboration during shopping (using the modesof operation of a shopping trip) can be used not only for product orcatalog or mall browsing but with any shopping facility or shopping toolsuch as the shopping cart, fitting room, wardrobe, user model,consultant, etc. Tools present in toolbar 239 such as editing zooming,panning, tilting, manipulating view, undo, etc, as described withreference to FIG. 20 can also be used during a shopping trip.

Reference is now made to FIG. 42 where one form of interaction betweenvarious parties with system 10 is shown in exemplary embodiment.Consumers can interact with their various computing devices 14, 16 notshown in the image. Other users may include shipping and handling users,administrative staff, technical support, etc. Consumers browse products,interact together and shop. When a purchase order is received at theportal server 20, vendors selling the product are notified. They thenapprove the purchase order, upon which the payment received from thecustomer is deposited in the corresponding vendor's account. Theshipment order is placed through shipping and handling users.Alternatively, the customer may pick up order at a store branch using a‘pick up ID’ and/or other pieces of identification. The store thecustomer is interested in picking up the order at can be specifiedthrough the system. The system may find the vendor store closest inproximity to the customer's location (customer's home, office etc.). Aninterface exists for interaction between any type of user and system 10,and between different groups of users via system 10. For instance,customers may interact with each other and with store personnel/vendors,and with fashion consultants via a webpage interface. Vendors mayinteract with customers, consultants and other businesses via a‘MyStore’ page available to vendors. Vendors can upload store feeds (inaudio, video, text formats etc.), product information and updates viathis page, as well as interact with customers. Vendors can see (limitedinformation on) who is entering their store in real time and alsooffline. For example, they can see if a set of users entering theirstore are on the same shopping trip, the age group of users (arbitrarynoise may be added to the age), the gender of the user. This allows thevendor to make comments like, “Hello boys, can I help you withanything?”. Users can set the privacy level they are comfortable withthrough the preferences panel. Fashion consultants can upload relevantinformation through pages customized to their need. They can upload thelatest fashion tips, magazines, brochures, style information etc. Theycan easily pull up and display to the user product information, dress‘how-tos’, style magazines and related information as appropriate. Theycan also interact via various forms of interaction (such asaudio/video/text chat etc.) described in this document.

Users on a shopping trip have the opportunity to use the Split-Bill™feature to make payments for purchases. Split-Bill is a feature thatenables users to share the cost of a purchase or the amount of atransaction by allocating some or all of the cost or amount to be paidby each of the users. Optionally, a subset of users that are party tothe transaction may be allocated the entire cost or amount of thetransaction. This feature also calculates the portion of taxes paid byeach individual in a transaction and can be used in conjunction with thereceipt management system discussed with reference to FIG. 48D.Split-Bill also enables users to claim their portion of an expense whenclaiming reimbursement for expenses (for example, expenses incurred onpart of an employee for the purposes of work). There are many optionsfor ways of operation of the Split-Bill feature. Most of these ways canbe thought of as similar to the modes of operation of a shopping trip asdescribed with reference to FIG. 7A-D. Some of these methods aredescribed next in exemplary embodiments: FIG. 21A demonstrates anexemplary embodiment of Split-Bill 261. Different payment schemes areavailable to the users of a shopping trip. A member of the shopping tripmay pay for the entire bill using option 262 or each member pay forhis/her individual purchases using option 263. Alternately, the bill maybe split between members by amount or percentage (as illustrated in FIG.21A) or other means of division using option 264. Such a service wouldalso be applicable to electronic gift cards available through system 10.More than one user may contribute to an electronic gift card and thegift card may be sent to another user via system 10. The recipient ofthe gift card would be notified by an email message or a notificationalert on his/her profile page or other means. The senders of the giftcard may specify the number of people contributing to the gift card andthe exact amount that each sender would like to put in the gift card orthe percentage of the total value of the gift card that they would liketo contribute to. In one exemplary embodiment, the Split-Bill methodworks as follows: When a user decides to split a bill on a supportedwebsite or application, they choose the friends that they wish to splitthe bill with and the portions of the bill that each friend includingthemselves will pay. After that, they confirm their order as usual andget sent a payment processing gateway to make payment. Once they havepaid their portion of the bill, the other participants are notified ofthe split bill payment. These other users accept the split billnotification and are sent to the confirmation page for an order wherethey confirm their portion of the bill and are sent to the paymentprocessing gateway. Once each member of the split bill group has madetheir payment, the order's status is changed to paid and becomes readyfor fulfillment. A hold may be placed on authenticated payment until allother participants' payments have been authenticated at which point allthe authenticated payments are processed. If a participant declines toaccept a payment, then the payments of all other participants may berefunded. Users can also split a bill with a friend (or friends) who isoffline. In this case, a user or users come to the Split-Bill screen andindicate the name of the user(s) that they would like to split a portionor all of the bill with. That user(s) is then sent a notification (onour website or on any other social networking site like Facebook or on achat application such as msn or via email or on a cell phonecommunicating via text such as through SMS or via voice by employingtext to speech conversion, in exemplary embodiments). That user(s) canthen decide to accept it in which case the transaction is approved andthe payment is processed, or deny it in which case the transaction isdisapproved and the payment is denied. This mode of operation is similarto the asynchronous mode of operation as discussed with reference toFIG. 7B.

In another exemplary embodiment, the Split-Bill method works as follows:When members of a shopping trip decide to split a bill on a supportedwebsite or application, each of them is presented with a screen such asthe one shown in FIG. 21B in an exemplary embodiment. In the first(leftmost) column, the user enters the amount that he/she would like topay (top row) of the total amount. Other users are shown similarscreens. As the user enters this amount, it is “flooded” (viewable) tothe other users' screens. The user can also enter the amount that he/shewould like other members to pay in the first column. The other columnsindicate the amounts that others have entered. For example, in FIG. 21Bit is shown that Alisha has entered “50” as the amount that she wouldlike to pay. In the 3-by-3 matrix shown, each column is for entering theamount that a member of the trip would like the members of the trip topay. A user (user A) can optionally override the amount that anotheruser (user B) should pay in their (user A's) column in the row thatcorresponds to the user's (user B) name. If the amounts entered by allthe members for any given row are consistent, a check mark appears. Inan exemplary embodiment, a user must enter the value in at least theirfield and column to indicate approval. The user cannot override thevalues in the grayed out boxes as these boxes represent the valuesentered by other users. If there is inconsistency in the values enteredin any row, a cross appears next to the row to indicate that the valuesentered by the users don't match. As the users enter their amounts an“Adds up to box” indicates the sum of the amounts that the users'contributions add up to. In an exemplary embodiment, the amounts alongthe diagonal are added up in the “Adds up to box”. Another fieldindicates the required total for a purchase. Yet another field shows howmuch more money is needed to meet the required total amount. If all rowsare consistent, the users are allowed to proceed with the transaction byclicking on the “continue” button. The amounts entered can be theamounts in a currency or percentages of the total. In an exemplaryembodiment, users can also view a total of the amounts that each of theusers is entering, as shown in FIG. 21C in an exemplary embodiment.Users can also select a radio button or a check box below the columncorresponding to a user to indicate that they would like that user'sallocation of amounts across friends. For example, as shown in FIG. 21Cthe user has chosen Alisha's way of splitting the bill. If all memberschose Alisha's way of splitting the bill, then a check mark appearsbelow Alisha's column and the users are allowed to proceed by clickingon the “continue” button. The user whom other members are choosing forsplitting the bill may also be communicated for example using colours.This mode of operation is similar to the synchronous mode of operationas discussed with reference to FIG. 7C.

In another exemplary embodiment, the Split-Bill method works as follows:When members of a shopping trip decide to split a bill on a supportedwebsite or application, each of them is presented with a screen such asthe one shown in FIG. 21D in an exemplary embodiment. Users can enterthe amount that they would like to pay in a field next to their name. Ifthe amount adds up to the required total, the users are allowed tocontinue with the purchase.

In another exemplary embodiment, the Split-Bill method works as follows:When members of a shopping trip decide to split a bill on a supportedwebsite or application, each of them is presented with a screen such asthe one shown in FIG. 21D in an exemplary embodiment. Users can enterthe amount that they would like to pay in a field next to their name. Inthis case, the users can enter an amount in any of the fields next tothe members names simultaneously using the communication protocoldescribed with reference to FIG. 7D. The users also share the same view.Each user also gets to approve his/her amount by checking a box next totheir name. If the amount adds up to the required total and each of theusers has approved his/her amount, the users are allowed to continuewith the purchase. This mode of operation is similar to the common modeof operation as discussed with reference to FIG. 7D.

During a shopping session, individual shopping carts as well as sharedshopping carts are available. In an exemplary embodiment, changes madeby a user of the shared shopping cart are synchronized across all usersof the shared shopping cart. An alternative option would be to make theshopping cart only viewable to others (read-only). Split-Bill alsoenables product-wise division. Users can also pick and choose whichitems from each of the members shopping carts they would like to payfor. An exemplary embodiment of such a method is illustrated in FIG.21E. As shown in this figure, a user has chosen to pay for his “RedJersey”, Alisha's sweater, and Robin's socks and tuque. The user's totalis also shown. Items that are paid for are shipped to the respectiveusers (shopping cart owners) or can be shipped to a common address(common to all users). Reference is now made to FIG. 21F where anotherexemplary embodiment of Split-Bill is shown. Users can drag and dropitems from a shared shopping cart into a list under their name. The listindicates the items that the user would like to pay for. At the bottomof the list the total of each user is also shown. Reference is now madeto FIG. 21G where another exemplary embodiment of Split-Bill is shown.Users can drag and drop items from a shared shopping list into a listunder their name and indicate the amount of the total bill that theywould like to pay. This could be an amount in a currency or a percentageof the bill. In another exemplary embodiment, users can state an amountor a maximum amount (which could even be zero) that they can afford topay. Other users can make payments on behalf of this user.

The Split-Bill feature can also work in any combination of the methodsdescribed above. In the above embodiments of Split-Bill, options arealso available to split a bill evenly between users or to split theoutstanding or remaining amount evenly between users. The aboveembodiments of Split-Bill can also be used in conjunction with multipleshopping trips. A trip leader may also be assigned to decide on how thebill is split. Reoccurring or monthly payments may also be sharedbetween friends using the above methods. This can also take place in around Robin fashion where one user pays the first month, a second userthe second month and so on. The Split-Bill feature allows processing ofcredit, debit, points cards and/or other supported payment options.Payments can be made using any combination of these options. Forexample, a product that is about to be purchased may be paid forpartially from a debit/bank account, partially via a credit card,partially using a gift card, and partially using points or storecredits. Points or credits may come from stores or from a user'sfriends. Also supported is the borrowing/lending of money and pointsbetween friends. This can be used in conjunction with contractmanagement system. The Split-Bill feature enables currency conversion.Users in different countries can view the amount to be shared in theirlocal currency or other currencies of their choice. The Split-Billfeature also enables users to request money or points from their friends(including those on social networks such as Facebook) or other users.This can be done when the user from whom money is being requested isonline or offline similar to the method described above. Upon approvalmoney or points get transferred to the account of the user who requestsfunds. This can then be transferred to the user's debit account, creditaccount, points account, etc. The amount of a transaction may also besplit between companies and other groups. For sites that do not supportthe Split-Bill feature, two or more parties can deposit to an accountusing the Split-Bill service on a supported site, upon which a debit ora credit or a points card or an electronic money voucher is created.This account can then be used on a third party site for a sharedpurchase. In an exemplary embodiment, the Split-Bill method is alsoavailable as an independent component on a website for people to sharethe amount of a translation. Users can collaboratively buyproducts/services and send them as a gift to other users. Users can alsoship gifts to users based on their location as specified in socialnetworking sites or on our site or based on their mobile devicelocation. This allows users to send gifts to an up-to-date address ofthe users' friends.

Investments may be made through Split-Bill. Other financial transactionsmay be conducted in a collaborative manner, including currency exchange.Currency may be exchanged, in exemplary embodiment, with a friend orsomeone in a friend's network so that the user may ensure that thetransaction is being carried out through a trusted reference. A persontraveling to another country may exchange money with a relative orfriend in that country. In another exemplary embodiment, shares andstocks may be traded collaboratively, for example through a split billinterface. Tools may be available for investors to collaboratively makeinvestments and assist them in making decisions.

Reference is now made to FIG. 35 where a virtual model is shown indisplay windows illustrating examples of how a user can animate theircharacter model's expressions/movements/actions and/or change theirmodel's look. The expressions/actions/dialogue/movements of thecharacter model can be synchronized with the user's ownexpressions/actions/dialogue/movements as tracked in the image/video (inan exemplary embodiment using a method similar to [52]) of the user orthese can be dictated by the user through text/speech and/or othercommand modes or through pre-programmed model expression/action controloptions provided through system 10. The display window 682 shows thevirtual model ‘raising an eyebrow’; display window 684 shows the modelwith a surprised expression sporting a different hairstyle; displaywindow 686 shows the virtual model under different lighting conditionswith a different hair colour. The exemplary embodiments in the figureare not restrictive and are meant to illustrate the flexibility of thevirtual models and how a user can animate and/or control their virtualmodel's looks, expressions, actions, background/foreground conditionsetc. Facial expressions may be identified or classified using techniquessimilar to those used in [53]. The virtual model can be thus manipulatedeven when the user uses it to communicate and interact with other users,for example, as in a virtual chat session. In another exemplaryembodiment of collaborative interaction involving a user's model,stylists and friends of the user can apply makeup to the user model'sface to illustrate make up tips and procedures. The makeup may beapplied to a transparent overlay on top the content (user model's face)being displayed. The system allows the user to save the animation andcollaboration sessions involving the user model.

Reference is now made to FIG. 36. This figure, in an exemplaryembodiment, shows a sample virtual store window 690 involving virtualinteraction between the user and a sales service representative in areal jewelry store, and incorporating augmented reality elements asdescribed next. In this example, a sales representative 691 interactswith the user in real-time via streaming video (acquired by a webcam orsome other real-time video capture device). The user in this instanceinteracts with the sales personnel via the user model 650 which islip-syncing to the user's text and speech input. Panoramic views of thedisplays 692 in the real jewelry store appear in the store window 690.An ‘augmented reality display table’ 693 is present on which the salesrepresentative can display jewelry items of interest to the user.Virtual interaction takes place via plug and play devices (for exampleI/O devices such as a keyboard, mouse, game controllers) that controlthe movement of simulated hands (of the user 694 and sales personnel695). Additionally, a device that functions as an ‘articulated’ controli.e., not restricted in movement and whose motion can be articulated asin the case of a real hand, can be used to augment reality in thevirtual interaction. Store personnel such as sales representatives andcustomer service representatives are represented by virtual charactersthat provide online assistance to the user while shopping, speak andorchestrate movements in a manner similar to real store personnel andinteract with the user model. The augmented reality display table isfeatured by system 10 so that vendors can display their products to thecustomer and interact with the customer. For example, a jewelry storepersonnel may pick out a ring from the glass display for showing theuser. A salesperson in a mobile phone store may pick out a given phoneand demonstrate specific features. At the same time, specificationsrelated to the object may be displayed and compared with other products.Users also have the ability to interact with the object 696 in 2D, 3D orhigher dimensions. The salesperson and customer may interactsimultaneously with the object 696. Physics based modeling, accomplishedusing techniques similar to those described in [54], is incorporated(these techniques may be utilized elsewhere in the document wherephysics based modeling is mentioned). This display table can be mappedto the display table in a real store and the objects virtually overlaid.A detailed description 697 of the object the user is interested in isprovided on the display while the user browses the store and interactswith the store personnel. A menu providing options to change settingsand controls is available in the virtual store window, by clicking icon540 in an exemplary embodiment. The above example of a virtual storeillustrates features that make the virtual store environment morerealistic and interaction more life-like and is described as anexemplary embodiment. Other manifestations of this virtual store may bepossible and additional features to enhance a virtual store environmentincluding adding elements of augmented reality can be incorporated.

Reference is now made to FIG. 22, where an apparel display window 400 isshown in an exemplary embodiment. The display windows provide visualrepresentations of the apparel items that are available tomodel/purchase to the user. The display window 400 comprises a visualrepresentation 402 of the apparel item. In the example provided herein,a visual representation of a skirt is provided. Further informationregarding the pricing, and ordering information, should the user desireto purchase this item is available. The user is able to view reviews ofthis apparel items that have been submitted by other users by engagingthe review icon 404 in an exemplary embodiment. The user is able tofurther share this particular apparel item with friends by engaging theshare icon 406 in an exemplary embodiment. If the user is browsing inthe regular mode of operation (not on a shopping trip with friends),clicking on this icon presents the user with a screen to select a modeof operation. If the synchronous mode or the common mode of interactionare chosen, the user is presented with a shopping trip window asdescribed with reference to FIG. 40. If the user chooses theasynchronous mode of operation, the item gets added to the “shareditems” list. The user can manage shared items through an interface asdescribed with reference to FIG. 23. If the user is engaged in thesynchronous or common modes of interaction, clicking on the icon 406,adds the item to the “shared items” list. The user can also send thisitem or a link to the item to users of social networking sites. The useris able to try on the apparel items on their respective user model byengaging the fitting room icon 408 in an exemplary embodiment. Themethod by which a user may try on various apparel items has beendescribed here for purposes of providing one example of such a method.Suitability of fit information may be displayed next to each catalogitem. In an exemplary embodiment, this is done by stating that the itemfits (‘fits me’) 410 and/or placing an icon that conveys the fit info(for eg. icon 550). Further details of displaying the goodness of fitinformation is described with reference to FIG. 30. A 2D or 3Dsilhouette 554 may also be placed next to catalog items to visually showgoodness of fit. Information on how the apparel feels is alsocommunicated to the user. This is done in an exemplary embodiment, bydisplaying a zoomed in image of the apparel 412 (“Feels Like”)illustrating the texture of the apparel. The sound that the apparelmakes on rubbing it may also be made available.

Models of products (photorealistic 3D models or NPR models) for use incatalogs may also be constructed by using images submitted by users.Images contributed by several users may be stitched together to createmodels of products. Similarly, images from several users may also beused to create a user model for the users' friend. Holes or missingregions, if any, present in the constructed models may be filled withtexture information that corresponds to the most likely texture for agiven region. The most likely texture for any given region can beestimated, in an exemplary embodiment, using Naïve Bayes or KNN. Thiscan be done as described earlier, using statistics drawn from regions inimages surrounding the holes as the input and the texture in the missingregion as the output.

When a user has chosen to try on an apparel items, the user is presentedwith a list of the various apparel items that have selected to try on inan exemplary embodiment. Reference is now made to FIG. 24, where asample fitting room window 420 is shown in an exemplary embodiment. Thefitting room window 420 lists the various apparel items that the userhas selected to try on. Each apparel item has an identification numberassigned to it by system 10 for purposes of identification. By selectingone of the items from the selection window 422, and clicking on icon424, the user requests that the system 10 fit and display the apparelitem on the user model. The status bar 426 displays the command that isexecuted—“dressbot:tryon=30” indicating that the item with ID(identification number) equal to 30 is being fitted on the user model.

An item of apparel is comprised of patterns (tailoring, stitch-and-sewterminology). All items of apparel are described that are associatedwith the system 10 have an apparel description file (ADF) associatedwith them. In exemplary embodiment, the ADF file can be in XML formatand the CAD file provided to system 10 by the retailer module 58 can beencapsulated within this ADF file. The apparel description file containsall information regarding the apparel including information necessary tomodel and display the apparel and to determine its fit on a model. Thisincludes, for example, the pattern information for a given apparel; howthe individual components of the apparel are stitched together; materialproperties such as composition, texture, etc; cloth care instructions;source information (country, manufacturer/retailer); optical propertiesincluding BDRF (Bidirectional Reflectance Distribution Function), bumpmap etc; microscopic images to reveal texture; location of where eachpiece goes with respect to anatomical landmarks on models. Any and allinformation related to the actual apparel and any and all informationneeded by system 10 to create the virtual apparel, display and fit it ona model is contained within the ADF file. An ADF file in XML format ispresented in FIG. 37 in an exemplary embodiment, The ADF file 700contains header information 701 followed by information describing aspecific apparel. The apparel tags 702 indicate the start (<apparel>)and end (</apparel>) of apparel description. Specific tags are providedwithin this region for describing different aspects of the apparel. Forinstance, the manufacturer description 703 includes the name of themanufacturer, the country source, the composition and size informationin this file. The care information 704 provides details on whether theapparel can be washed or dry-cleaned; the pattern tags 705 enclose theCAD filename containing the details on apparel pattern data; the fittinginformation 706 that describes how a virtual manifestation of theapparel fits on a virtual human model is encapsulated by the fittingtags 706; the media tags 707 enclose filenames that provide visual,audio and other sense (such as feel) information about the apparel, aswell as the files and other data containing display information aboutthe specific apparel (the 3D display data for the apparel model lieswithin the <render> tag in this example). Further store information 708such as the unique store ID in the system 10, the name of the store andother details relating to a specific store such as the return policy isprovided in the ADF file. The ADF file 700 in FIG. 37 is presented forpurposes of illustration and is not meant to be restricted to the XMLformat or the tags given in the file. Other manifestations of the ADFare possible and other tags (descriptors) may be included to describe agiven apparel. Much of the information describing the apparel iscontained in the CAD file obtained from the retailer 58, while theinformation necessary to model, display and fit the apparel is augmentedwith the CAD file to form the ADF. Reference is now made to FIG. 38where a quick overview is provided of ADF file creation and use, in anexemplary embodiment. Apparel information 711 described previously, aswell as information associated with the specific apparel in its CAD fileis packaged by the ADF creation software 712 to form the ADF file 700.This ADF file information is then subsequently used in modeling theapparel digitally for purposes of display in electronic catalogues anddisplays 713; for fitting on 3D user models 714; for displaying andlisting in the virtual wardrobe and fitting room 715 as well as otherforms of digital apparel viewing and interaction. Pattern informationcomprising the apparel is extracted. This information is contained inthe CAD and/or ADF files and is parsed to form the geometric and physicsmodels of the apparel. In forming the geometric model, a mesh isgenerated by tessellating 3D apparel pattern data into polygons. Thisgeometric model captures the 3D geometry of the apparel and enables 3Dvisualization of apparel. The physics model is formed by approximatingthe apparel to a deformable surface composed of a network of pointmasses connected by springs. The properties of the springs (stiffness,elongation, compressibility etc.) are adjusted to reflect the propertiesof the material comprising the apparel. The movement of the cloth andother motion dynamics of the apparel are simulated using fundamentallaws of dynamics involving spring masses. Cloth dynamics are specifiedby a system of PDEs (Partial Differential Equations) governing thesprings whose properties are characterized by the apparel materialproperties. The physics model enables accurate physical modeling of theapparel and its dynamics. Reference points on the apparel specifyregions on the apparel corresponding to specific anatomical landmarks onthe human body. The information concerning these points and theircorresponding landmarks on the body will be contained in the CAD and ADFfiles. The reference points on the geometric and physics based models ofthe apparel are then instantiated in 3D space in the neighbourhood ofthe corresponding anatomical landmarks of the character model. Fromthese initial positions, the reference positions are pushed towards thetarget anatomical positions. At the same time, springs interconnectingseams are activated to pull together the simulated apparel at the seams.FIG. 29A illustrates an example of the visual sequences 460, from leftto right, displayed to the user in a window while the apparel is beingfitted on a non photorealistic rendering of the user model. An exampleof the visual sequences 462, from left to right, presented to the userin a window during hair modeling on the non photorealistic rendered usermodel is also shown in FIG. 29A. The hair 464 on the user model isanimated using physics-based techniques which permit realisticsimulation of hair look and feel, movement and behavior.

Reference is now made to FIG. 29B where a user model adjustmentsinterface 470 is shown in an exemplary embodiment, containing a nonphotorealistic rendering of a user model. Options to make bodyadjustments are displayed upon clicking the menu display icon 476. Asample mechanism is shown for making adjustments to the body. Slidercontrols 475 and 477 can be used to make skeleton and/or weight relatedadjustments to the user model. Skeleton adjustments allow modificationsto be made to the generative model of the skeletal structure of the usermodel. This renders anatomically accurate changes to be made to the usermodel. In an exemplary embodiment, upon moving some of the skeletonadjustment controls 475 to the right, a taller user model (withelongated bones) 472 is obtained whereas, by moving some of the skeletonadjustment controls 475 to the left, a petite user model 471 isobtained. In another similar exemplary embodiment, weight adjustmentcontrols 477 can be used to obtain a heavier user model 474 or a slimmeruser model 473. In an exemplary embodiment, manipulating the skeletaladjustment controls increases or decreases the distance between a jointand its parent joint. For example increasing the value of the length ofa shin increases the distance between the ankle joint and its parentjoint, the knee joint. In an exemplary embodiment, manipulating theweight adjustment controls increases or decreases the weight assigned tothe corresponding vertices and moves them closer or farther from theskeleton. For example, increasing the weight of a selected portion ofthe shin places the vertices corresponding to that region further fromthe skeleton. Continuity constraints (a sigmoid function in an exemplaryembodiment) are imposed at the joints to ensure plausible modificationsto the user model. Users can also deform the user model by nudging thevertices corresponding to the user model. Users can also specify thebody muscle/fat content which sets the appropriate physical properties.This is used, for example, to produce physically plausible animationcorresponding to the user.

Reference is now made to FIG. 29C where a sample window is showndemonstrating product catalogue views available to the user from whichapparel may be selected for fitting onto their user model. A productcatalogue 480 may be displayed by clicking a menu display icon 482. Theuser may then select a given outfit/apparel/product from the catalogueupon which it will be fit and displayed on the user model. In exemplaryembodiments, product catalogues are available in the local application271 or within the browser or a combination of both as described withreference to FIG. 10 and FIG. 31.

By clothing the user's model with apparel chosen by the user, the useris able to visualize and examine the appearance of the apparel on theirbody from an external perspective and also get an approximate idea ofhow the apparel fits. In order to communicate fit information to theuser in more exact terms, metrics are used that define the suitabilityof apparel not just based on size information but also as a function ofbody type and fit preferences. The system will relay suitability of fitinformation to the user using aspects that are, but not limited to,quantitative and qualitative in nature. For example, goodness of fit isa quantitative metric. In exemplary embodiment, for determining apparelgoodness of fit on a user model, the convex hull of the model iscompared with the volume occupied by a given piece of clothing. Asmentioned previously, apparel can be modeled as springs by system 10. Inorder to determine regions of tight fit in this case, in exemplaryembodiment, physical stress and strain on the apparel and/or model canbe computed using the spring constant of the apparel material. Regionsof loose fit may be determined by evaluating normals from the surface.The distance between the body surface and the apparel surface can beascertained by computing the norm of the vector defined by theintersection of the surface normal to the model's surface with the clothsurface. This process can be made computationally efficient by samplingsurface normals non-uniformly. For instance, regions of high curvatureand greater importance may have many more normals evaluated than regionsof low curvature. In assessing suitability of fit, qualitative aspectsare also incorporated by system 10. These include, but are not limitedto, user preferences. An example of this is the user preference forloose fitting clothes. On their user model, users can visualizesuitability of fit through various visualization schemes provided bysystem 10. In exemplary embodiment, regions of different fit on theapparel may be colored differently. Visual indicators include, but arenot limited to, arrows on screen, varying colors, digital effectsincluding transparency/x-ray vision effect where the apparel turnstransparent and the user is able to examine fit in the particularregion. Some examples are illustrated in FIG. 30. The visualizationoptions are provided to the user via a menu available by clicking theicon 540, in exemplary embodiment. In this figure, different fit regionsare depicted using coloured arrows 542, highlighted regions 544 as wellas transparency/x-ray effects 546. Transparency/x-ray effects 546 allowfit information to be visualized with respect to body surface. In FIG.30, the apparel on the 3D body model is made transparent in order forthe user to visually examine overall apparel fit information—regions oftight/proper/loose fit. With reference to FIG. 30, regions of tight fitare shown using red coloured highlight regions (armpit region). Loosefitting regions are shown via green arrows (upper leg) and greenhighlight (hips). Comfort/smug fitting is depicted using orange arrows(waist) and yellow highlight (lower leg). Users may also define thenumerical margins that they consider ‘tight’, loose’ and so on fordifferent apparel. For example, the user may consider a shirt to beproper fitting around the arms if the sleeves envelope the arm leavingbetween 1-2 cm margin. The user may specify these margins and othersettings using the options menu 540 available to the user. Thetransparency/x-ray effect also provides visual information with regardsto layers of clothing. The users may wish to select particular items forvisualization on the model. In one exemplary embodiment, they may selectfrom the itemized list 552 which lists all of the apparel items the userhas selected to fit on the user model as part of an ensemble forinstance. Accordingly, the items that are not selected may disappear orbecome transparent/light in colour (i.e., recede or fade) in order tomake more prominent the selected items of apparel. Thus, thetransparency effect emphasizes certain items visually while stillpreserving other layers of clothing so that the highlighted apparel maybe examined with respect to other items it will be worn in combinationwith. The layers worn by the model in FIG. 30 may be examined fromdifferent perspectives of the model (cross-sectional view for example).This page also provides the user with the menu (available by clickingicon 540) described previously for setting/manipulating the model andenvironment as well as setting view options, share options (for example,sharing model views with friends in specific apparel). Other purposesfor which visual indicators may be applied includes, but is not limitedto, relaying the user with information regarding the quality or make ofan apparel. For example, different colours may be used to outline orhighlight a shoe sole in order convey whether the given shoe ishard-soled or soft-soled, Separate icons may also be provided such as548 provided to interact and/or manipulate model as shown in FIG. 30.Additionally, an icon summarizing suitability of fit may be provided550. This will incorporate all the quantitative and/or qualitativeaspects assessing goodness of fit and give the overall consensus onwhether the apparel will fit the user (thumbs up) or not (thumbs down)in an exemplary embodiment. The ‘summary’ icon may be programmed bydefault, for example, to give a ‘thumbs up’ if two qualitative andquantitative aspects are satisfied. This default setting may be changedto suit the user's suitability of fit requirements. More details on thefit are available to the user by clicking on or hovering over the icon550. The user can also choose to display portions of these details nextto the icon through the preferences page. In an exemplary embodiment,the user can see the fit information by taking an item to the fittingroom (eg. by dragging and dropping a catalog item into the fittingroom). In another exemplary embodiment, the user can see all the itemsthat the user is browsing with the fit information without the need toplace the item in the fitting room. All instances of features shown inFIG. 30 are illustrative examples and are not meant to be restricted tothese and can embody and encompass other forms, illustrations andtechniques.

Reference is now made to FIG. 23, where a sample shared item window 430is shown in an exemplary embodiment. The shared item window 430 displaysthe various items that the user has shared, in a shared list 432, and alist of items that friends have shared in a friend shared list 434. Thesnapshots lists 436 allow a user to share various images that they havecaptured of their user model with other users. When viewing andinteracting with the user model, the user is provided the ability tocapture an image or snapshot of the image, and share the respectivesnapshot or image with other users. These features illustrate oneexemplary embodiment of the asynchronous mode of operation of a shoppingtrip.

Reference is now made to FIG. 25, where a sample wardrobe image 440 isshown in an exemplary embodiment. Wardrobe images 440 are used in anexemplary embodiment to display to the user the apparel items that auser has added to their wardrobe. A user may browse all of the itemsthat are in their virtual wardrobe, and may request that they receivecomments regarding items in their wardrobe from a consultant. The useris presented with options as in the tabbed menu 442 shown in exemplaryembodiment, so that they can quickly navigate and browse the apparel intheir wardrobe and fitting room; try on apparel on their model as wellas get feedback regarding apparel and dressing style options from thestyle consultant. From left to right, the icons 444 available to theuser in their wardrobe include: (1) the icon that displays to the userapparel information such as the make and manufacturer details, careinstructions, store it was bought from, return policy etc. as well asuser tagged information such as who gifted the apparel, the occasion towear it for, etc.; (2) the icon to fit selected apparel on the usermodel; (2) the icon to share selected apparel with other users. Theicons shown have been presented as examples and may include icons thatperform other functions. The icons shown may be represented withdifferent symbols/pictures in other manifestations. Reference is made toFIG. 28 where a drawing of a 3D realization of a virtual wardrobe isshown. This wardrobe can be incorporated with physics based animationfunctionality so that users can drag around objects; arrange and placethem as desired in the wardrobe; move them into boxes or bins or hangersor racks etc. Users will be able to visualize articles of clothing andother apparel in their wardrobe; tag each item with a virtual label thatmay contain apparel specific information as well as user specifiedinformation such as the date the apparel was bought; the person whogifted the apparel; upcoming events on which it can be worn as well aslinks to other items in the wardrobe and/or fitting room with which thatitem can be coordinated or accessorized with etc. Reference is made toFIG. 26, where a sample style consultant window 450 is shown in anexemplary embodiment. The style consultant 452 is able to comment on theuser's items in the wardrobe, upon request of the user. The icons 454shown from left to right include: (1) the icon to obtain information onthe specific style consultant; (2) the icon to add/remove styleconsultants from the user's personal list. Icon 456 provides the userwith options to engage in communication with the style consultant eithervia email or chat which may be text/voice/video based or may involveaugmented reality, in exemplary embodiments.

Reference is now made to FIG. 27 where a sample diagram is presentedillustrating the actions involving the fitting room 420 and wardrobe 440that the user may engage in while browsing for apparel. While browsingfor apparel displayed as in example window 400, the user can add an itemto their fitting room by clicking on an icon 424 next to the item theywish to virtually try on. Once an item has been added to the fittingroom 420, that item will become available to the user in the localapplication 271 for fitting on their model. Once the item has been addedto the fitting room, the user may model the apparel item on their usermodel, and/or decide to purchase the item, in which case the apparelitem can be added to the virtual wardrobe 440. Alternately, the user maydecide not to purchase the item in which case the item will stay in thefitting room until the user chooses to delete it from their fittingroom. The user may choose to keep a purchased item in their wardrobe 440or delete it. If the user decides to return an item, that item will betransferred from the user's wardrobe 440 to the fitting room 420. Theuser may also decide to conduct an auction or a garage sale of some orall of the real items in their wardrobe. Users with access to thevirtual wardrobe can then view and purchase items on sale of interest tothem via system 10. The virtual items in the fitting room and wardrobecan also be purchased for use in other sites that employ virtualcharacters/models. The virtual apparel items in the fitting room andwardrobe may be exported to external sites or software involving virtualcharacters/models such as gaming sites, ‘virtual worlds’ sites andsoftware.

Reference is now made to FIGS. 46A to 46H where other exemplaryembodiments of the features described in this patent have beenpresented. FIG. 46A shows a profile or home page of a user registeredwith system 10. The user can grant access to this page to other users bysetting permissions. A master menu 800 with option tabs—‘profile’,‘browse’, ‘shopping trip’, ‘cart’, ‘shopping diary’ is shown at the topof the page. These tabs navigate to pages which allow the user torespectively, access their profile page; browse stores and products;manage collaborative shopping trips; view and manage items in cart;access personalized shopping and other miscellaneous features. Icon 801displays the logo of system 10 and provides the user with a menucontaining certain options such as home page access and help withfeatures available to the user on system 10. Display box 802 representsthe information card providing profile details of the user. Display box804 contains hyperlinks to all stores subscribing to system 10 or justthe favourite/most frequently visited stores by the user. Additionally,users may engage display box 805 for adding friends they would like tocollaborate with. In an exemplary embodiment, users may add friends theynormally like to acquire feedback from or go out with for shopping. Theuser may also add other users registered with system 10 whosefashion/style sense they like and follow (the user would be thatperson's ‘style fan’ in that case). Another menu 803 is provided in FIG.46A as an exemplary embodiment which permits the user to access morefeatures available on system 10.

Reference is now made to FIG. 46B where a store page 806 is shown. Theproducts available in the store 808 may be categorized according todifferent fields such as department, category, size etc. Users may alsobe able to search for products in the store. Stores have the option ofpersonalizing their store pages. In an exemplary embodiment, theseason's collection may be displayed in a product display window 809.Items featured by the store and other item collections may also bedisplayed in another window 810. FIG. 46B also displays a collaborativeshopping trip window 807 on the same page. The shopping trip window maybe launched by clicking on icon 815. The shopping trip dialog 807containing collaborative shopping features may open up in a separatewindow or in the same window/page being viewed by the user. Somecollaborative shopping features are illustrated in the shopping tripdialog 807 as exemplary embodiments. A synchronized product viewer 811enables collaborative shopping between members of that shopping tripdisplayed in window 814. Products being browsed by other users of theshopping trip may be viewed in the product viewer 811 via menu 812. Byselecting a given user in window 814, the user can browse the shoppingcart, shopping list, wishlist, wardrobe, and other personalized shoppingfeatures shown in 814 of the selected user, if that user has grantedpermission, by clicking on the ‘GO’ button in window 814. A chat window813 and/or other synchronous or asynchronous means of communication maybe available to enable communication with other users while shopping.Reference is now made to FIG. 46C which illustrates another layout inexemplary embodiment. This layout combines some store page features withcollaborative shopping trip features on the same page. A regular storepage 806 shown in FIG. 46B may convert to a page as in FIG. 46C uponactivating the shopping trip. Reference is now made to FIG. 46D where asample shopping trip manager window/page is shown. Users can create newshopping trips 816; categorize trips by labeling them and invite friendson shopping trips. Users can view and sort shopping trips 817 accordingto labels.

Reference is now made to FIG. 46E where a user's personalized ‘looks’window/page is shown in exemplary embodiment. A ‘look’ in this contextis defined as a collection of products put together by the user fromdifferent product catalogues to create a complete ensemble or attiredefining a suggested ‘look’. Other users may gauge a user's fashionsense or style by browsing through the given user's looks page. Abrowser window 818 allows the user to browse looks they created. Eachlook 819 is composed of several items put together by the user. In anexemplary embodiment, a look 819 may contain a blazer, a blouse, askirt, a pair of shoes, a handbag and other accessories to complementthe given look. A user may obtain expanded views of products comprisinga given look by highlighting a look 819, upon which another dialog orwindow 820 is launched containing expanded views 821 of items composing819. Upon selecting an item in the expanded view 820, a product optionsmenu 822 appears which is comprised mainly of the four option boxesoutlined in red. The other sub-menus 823-826 appear upon clicking therespective main product menu options besides which they appear. Theproduct options menu 822 is shown in exemplary embodiment and it enablestasks such as product purchase 824, product sharing with other users823, rating the product according to different criteria 825 and additionof the product to various personalized user lists 826.

Reference is now made to FIGS. 46F-G where other exemplary embodimentsof the fitting room window have been shown. FIG. 46F shows some featurescomprising the fitting room 827. These may include the shopping cart828, or items that the user has selected but is undecided aboutpurchasing 829, and the product viewer 830 which provides product viewsof the item selected from the shopping cart or the ‘decide later’ cart.Another version of the fitting room is shown in FIG. 46G whichincorporates the product viewer 830, the shopping cart, ‘decide later’items as well as other customized user lists such as shared items, toppicks, my looks and others.

Reference is now made to FIG. 46H where a shopping diary window/page andits features are shown in an exemplary embodiment. The shopping diary iscomprised of personalized user lists such as shopping lists, wishlists,gift registries, multimedia lists and others. Additionally it mayincorporate a shopping blog and other features.

Reference is now made to FIG. 46I where a layout or directory of themall comprising stores subscribing to system 10 is shown in an exemplaryembodiment. This can be customized to form a user-specific directorythat lists businesses and people that a user is associated with in acommunity. Stores are listed on the left and categorized by gender andage group. A map or layout 1106 of the virtual mall is presented to theuser where the stores on system 10 may additionally be shown graphicallyor using icons. Upon selecting a store 1100 from the list, a store image1104 may be displayed. A ‘window shopping’ feature permits users to getlive feed from the store including information 1105 such as other usersbrowsing the store. The user may be able to identify contacts in theirfriends list who are browsing the store via this feature and alsoidentify the contact's category (i.e., work—W, personal—P etc.).Additionally, other services 1102 may be listed such as dental and otherclinics. Users may be able to book appointments online via a clinicappointment system available through system 10. Users may also make useof a ‘smart check’ feature that checks the user's calendar for availableslots and suggests potential dates to the user for booking appointmentsand/or proceeds to book the appointment for the user by providing theclinic with the user's availability dates. Once the clinic confirms abooking, the smart check calendar feature informs the user of theconfirmed date via SMS/email/voicemail/phone call. Users may set theirpreferred method of communication. It may additionally suggest to theclinic the best dates for scheduling an appointment by cross-referencingboth the patient/client's schedule and the clinic's schedule. Users maymark other appointments in their digital calendar. The calendar may sendappointment reminders via SMS, email, phone call to the user dependingon user preferences and the user will be presented with options toconfirm, cancel or postpone the appointment upon receiving theappointment reminder. The calendar would notify the user of the durationafter which the appointment is scheduled, for example—‘your dentistappointment is in 15 minutes’. Furthermore, the smart-check featurecould also cross-reference the dentist clinic's electronic schedule inreal time and inform the user whether their appointment is delayed orpostponed because the clinic is not running late or for some otherreason. Other services such as food/catering 1103 may be availablepermitting the user to order online. Another feature available on system10 is an ‘electronic receipt manager’. This feature allows the user tokeep track of all receipts of products purchased through system 10 andother receipts that the user may want to keep track of. This may proveuseful to users for purposes such as exchanging or returningmerchandise, tax filing, corporate reimbursements and others. Userswould be able to categorize receipts (example, business, personal etc.);import and export receipts to other places such as the user's localcomputer or a tax filing software and other places; conduct calculationsinvolving amounts on those receipts. Stores on system 10 may also findit useful to have and store these electronic receipts in order tovalidate product purchases during a product return or exchange.(Receipts for purchases made at the physical stores can also be uploadedto the electronic receipt manager. This can also be done at the point ofsale (POS)). An interface for the Electronic Receipt Manager and furtherdetails are described with reference to FIG. 48D. The store and serviceslayout 1106, and store and services listing may also be customized bythe user to comprise favourite stores and services of the user i.e.,stores and services such as the dentist, mechanic, family physician,hair salon, eateries etc. most frequently visited by the user (may beentitled ‘My Business’ section in exemplary embodiment). This wouldpermit the user to create their own virtual mall or virtual communityproviding quick and easy access to stores and services most beneficialto the user as well as their contact and other information. (Users cansearch for businesses and add them to their ‘community’ or contactslist. On searching for a business using a name, a list of businesseswith that name or similar names may be shown and may be displayed inascending order of the distance from the user's home, office, city, orcurrent location). A user can also visit other users' virtual malls andcommunities. Alternatively, a virtual mall may be mapped to a real malland contain stores and services that are present in the real mall. Inexemplary embodiment, the ‘My Business’ concept described above may beintegrated with social networking sites. Tools may be available tobusinesses to communicate with the user clients and customers, such asvia the clinic appointment system described above. Tools may beavailable to customers to manage receipts, product information and alsoto split bills. The system described with reference to FIG. 46I may beintegrated with the VOS and/or VS described in this document.

Reference is now made to FIGS. 47 A-B which illustrate features thatallow the user to customize pages on system 10; to set the theme andother features that allow the user to personalize the browserapplication's and/or local application's look and feel. FIG. 47A shows atheme options menu 1108 where a user can choose and set the colour themeof the browser pages that they will be viewing during their session onsystem 10. In the instance shown in FIGS. 47A and 47B, the user haschosen ‘pink’. Accordingly, the theme changes as shown via the windowsin FIGS. 47A-B. FIG. 47B also shows features available to the user forspecifying the delivery information 1112 of a product upon purchase.Users may specify a friend from their address book or friends' list andalso specify the delivery location type (i.e., work, home etc.). Thesystem would then directly access the latest address information of thatfriend from their user profile. This address would subsequently be usedas the delivery address.

Reference is made to FIGS. 48A-F, where some features and layout designsof system 10 are illustrated in exemplary embodiment. These features anddesigns can be used with the local application or a web browser or awebsite in exemplary embodiments. The description of these figures isprovided with respect to the local application but it also holds in thecase of a browser implementation or a website implementation of thesame.

Reference is now made to FIG. 48A. The display screen 1130 is encased byan outer shell 1131, henceforth referred to as the ‘faceplate’ of thelocal application. The faceplate can be changed by a user by selectingfrom a catalogue of faceplates with different designs andconfigurations, which will be available under menu options.

On the faceplate are navigation links represented by buttons with icons1132, in an exemplary embodiment. The lifesaver icon 1133 serves as alink for the help menu. Button 1134 represents the user accountnavigation link which directs the user to their account or profilespace/section on the local application, consisting of the user'spersonal information, account and other information; settings andoptions available to the user to configure their local application orbrowser application; information and links to tools and applicationsthat the user may add to their local or browser application. Navigationlink 1135 on the faceplate is discussed with reference to FIG. 48A.Other navigation links on the faceplate will be discussed with referenceto the figures that follow. Button 1135 directs the user to the usermodel space/section of the local application (button 1135 is highlightedwith a red glow here to show that it is the active link in this figurei.e., the screen 1130 displays the user model space). In this space,users can access their 3D model 1136. Menu options 1137 for viewing,modifying and using the 3D model are provided on this page. Otherfeatures may be present in this space that can be utilized inconjunction with the 3D model. For instance, the fitting room icon 1138is provided as an exemplary embodiment. Upon activating this icon (byclicking it for example), the fitting room contents are displayed 1139(in the form of images here) enabling the user easy access to theapparel they would like to fit on their user model 1136.

Reference is now made to FIG. 48B. In this figure, navigation link 1145,which represents ‘shopping tools’ is shown as being active. Hence, inthis figure, the display screen 1130 displays the shopping tools spaceof the local application. This space provides the user with applicationsand options that assist in shopping online and/or electronically via thelocal application software. Most of these features have been describedpreviously in this document and are discussed here mainly to illustratean exemplary embodiment of the local application's layout. Icon 1146,when activated (by hovering over icon with mouse or by clicking icon, asexamples) displays a menu of user lists 1147 (shopping list, wishlist,registries etc.), which may be used to document shopping needs. Thismenu 1147 subsides/is hidden when the icon is deactivated (i.e., bymoving the mouse away from the icon or by clicking the icon afteractivating it, as examples). Icons 1148-1152 in FIG. 48B function in asimilar way in terms of activation and deactivation. Icon 1148 providesa menu with features to assist in shopping and in making the shoppingexperience immersive. As shown in the figure, these features include thecollaborative shopping trip feature, consultation (online or offline)with a style or fashion expert among others. Feature 1149 provides theuser with access to gift catalogues, gift cards/certificates, as well asinformation on gifts received and sent. Icon 1150 provides the shoppingcart menu listing items that the user has chosen for purchase; that theuser has selected for making a decision to purchase or not at a laterdate. It also directs the user to the checkout page. Feature 1151assists the user in making shopping related searches and also in seekingout products in specific categories such as ‘top bargains’, ‘mostselling, ‘highest rated’ etc. Icon 1152 provides features customizableby the user and/or user specific tools such as item ratings, producttags or labels etc.

Reference is now made to FIG. 48C. Navigation link 1160, whichrepresents the ‘connect’ feature is shown as being active. This linkdirects the user to the social networking space of the localapplication. The list box 1161 provides the user with a listing of theuser's friends and other contacts. It may contain contact names, contactimages, web pages, personal and other information relating to eachcontact. Feature 1162 provides the user with the facility to selectmultiple contacts (in this case, feature 1162 appears in the form ofcheckboxes as an exemplary embodiment). On the right side of the displayscreen 1130, social networking features are provided i.e., applicationsthat provide the facility to shop, communicate, interact online,virtually and/or electronically and perform other activitieselectronically with contacts. Some of these features are illustrated inFIG. 48C as an exemplary embodiment. Icons 1163, 1165, 1167 can beactivated and deactivated in a fashion similar to icons 1146, 1148-1152in FIG. 48B. Upon activating icon 1163, a shopping trip invite menu 1164appears, providing the user with options to send an automated oruser-customized shopping trip invitation message to all or selectedcontacts from the list 1161. These options are symbolized by the iconsin the menu 1164. From left to right, these icons allow the user to sendinvitations via ‘instant notification’, ‘phone’, email’, ‘SMS’ or ‘textmessage’, and ‘chat’. Feature 1165 provides a menu with options tocommunicate with all or selected users in 1161. These options aresimilar to the ones in menu 1164. Feature 1166 provides the user withgift giving options available on system 10. Users can select friends in1161 via 1162 and choose the from the gift options available in menu1167. From left to right in menu 1167, these icons represent thefollowing gift options: ‘gift cards’, ‘shop for gifts’, ‘donate withfriends’, ‘virtual gifts’. This list can contain other gift options suchas the ones provided by 1149 in FIG. 48B. The arrow 1168 allows the userto navigate to other applications in this space that are not shown herebut maybe added later.

Reference is now made to FIG. 48D. In this figure, the ‘financial tools’link 1175 is shown as active and the corresponding space that the useris directed to is shown in the display screen 1130. Some of the featuresaccessible by the user in this space are described next. Feature 1176and other icons in this space can be activated and deactivated in amanner similar to icons in other spaces of the local application, asexplained previously. Upon activating icon 1176, options menu 1177appears displaying options that can be used to view, manage and performother activities related to purchase receipts, refunds and similartransactions. Some of these are shown in 1177—‘billing history’ allowsthe user to view the complete listing of financial transactionsconducted through system 10; ‘pay bills’ allows the user to pay forpurchases made through system 10 via a credit card provided for makingpurchases at stores on system 10; ‘refunds’ assists in making andtracking refunds; ‘manage receipts’ allows the user to organize, labelelectronic receipts, and other housekeeping functions involving theirreceipts, perform calculations on receipts; ‘edit tags’ allows users tocreate, modify, delete receipt/bill tag or labels. These could include‘business’, ‘personal’ and other tags provided by the system or createdby the user. The accounts feature 1178 provides options that allow theuser to view and manage accounts—balances, transfers and other accountrelated activities, account statistics and other account specificinformation. These accounts can be mapped to a user's banking accounts,which may be at multiple financial institutions; these could includecredit/debit card accounts; accounts for credit cards provided forconducting financial transactions on system 10; gift card accounts.Feature 1179 provides other tools that assist the user in managingfinancial transactions conducted on system 10, as well as financialaccounts, and other personal and business finances. Some of these areshown in the figure and include—‘expense tracker’, ‘split bill’ whichwas described previously in this document, ‘currency converter’, taxmanager’ etc. Since this is a space requiring stringent securitymeasures, icon 1180 details the user on security measures taken bysystem 10 to protect information in this space. The electronic receiptsmay be linked with warranty information for products from themanufacturer/retailer, so that users may track remaining and applicablewarranty on their products over time. For the manufacturer and retailer,the electronic receipt information on a user's account may serve usefulfor authenticating product purchase and for warranty application terms.Since the receipt is proof of product purchase, it may also be used tolink a user's account containing the receipt for a product, with theuser manual, product support information and other exclusive informationonly available to customers purchasing the product. Other informationsuch as accessories compatible with a product purchased may linked/sentto the user account containing the product's receipt.

Reference is now made to FIG. 48E where the ‘share manager’ space (1185)on the local application is described. User files on a local machine oron in the user account on system 10 can be shared by activating a shareicon similar to 1186. Items may be shared in other spaces as well butthis space provides a comprehensive list of features for sharing items,managing shared items, users and activities involving shared items.Users can keep track of items they have shared with other users (1187,1188). Users may change share settings and options, view their sharingactivity history, tag shared items, add/remove files/folders and performother actions to manage their sharing activity and items (1189, 1190).Users may maintain lists of other users they share items with, subscribeto and send updates to sharing network on items shared, and maintaingroups/forums for facilitating discussion, moderating activities onshared items (1191).

Reference is now made to FIG. 48F where the ‘user model tools’ space isdescribed. Here the user can access make changes and manage their 3Dsimulated user model and model profile information (1212). Style toolsare available to assist users in making better fashion choices whileshopping for clothes and apparel (1214). These tools include consultingor acquiring fashion tips/advice from a fashion consultant, constructinga style profile which other users or fashion experts may view andprovide appropriate fashion related feedback. A ‘my look’ section isalso present in this space where users can create their ownensembles/looks by putting together items from electronic clothing andapparel catalogues (available from online stores for example). Further,users may manage browse or search for outfits of a particular style instore catalogues using style tools provided in this space (1214). Avirtual fitting room (1216) is present to manage apparel itemstemporarily as the user browses clothing stores. Apparel in the fittingroom may be stored for trying on/fitting on the user model. A virtualwardrobe space (1218) also exists for managing purchased apparel orapparel that already exists in the user's physical wardrobe. Thesimulations/images/descriptions of apparel in the wardrobe may becoordinated or tagged using the wardrobe tools (1218). The fitting roomand wardrobe feature and embodiment descriptions provided earlier alsoapply here.

Throughout the FIGS. 48A-F, the application has been referred to as a‘local application’. However, this application may also be run as awhole or part of a web application or a website or a web browser or asan application located on a remote server.

Operating systems need to be redefined to incorporate collaborativeenvironments and functions. Reference is now made to FIGS. 49A-O wherean immersive Application and File Management System (AFMS) or VirtualOperating System (VOS) and its features are described. This AFMS/VOSsystem or a subset of its features may be packaged as a separateapplication that can be installed and run on the local or networkmachine. It can also be implemented as a web browser or as part of a webbrowser and/or as part of an application that is run from a web serverand can be accessed through a website. It can also be packaged as a partof a specialized or reconfigurable hardware or as a piece of software oras an operating system. This application may be platform independent. Itmay also take the form of a virtual embodiment of a computing deviceshown in FIG. 2.

An exemplary embodiment of AFMS system and its features are described inFIGS. 49A-L. FIG. 49A is a login window that provides a layer ofsecurity which may or may not be present when an application using thissystem is accessed depending on the security level selected.

Reference is now made to FIG. 49B where some file category and searchlevel features are demonstrated, in an exemplary embodiment. Defaultfile categories may be provided with the system and are some are shownin the figure in an exemplary embodiment. These are folders to store weblinks (1250), shopping related content (1252), multimedia relatedcontent (1254) and data files (1256). Users may create their own foldersor remove any of the default folders provided, if they wish. In thisfigure, the shopping related folder is selected. It contains thecategories or tags 1258, which are shown in exemplary embodiment. Theuser can create new tags, remove tags, create sub-level tags/categoriesand so on. The user can also conduct tag-keyword specific files searcheswithin the system. For instance, the user can go the product tag andaccess the sub-tags (1260) within this category. The user can select thekeyword Canon200P (highlighted in orange in the figure). Othertags/sub-tags (1264) can be similarly selected to be used in combinationin the keyword specific search. An operator menu 1262 is provided sothat the user can combine the tags using either an ‘OR’ or ‘AND’operator in order to conduct their search, the results of which can beobtained by clicking the search operator 1266. The user may also chooseto filter certain results out using the ‘filter’ function 1268 whichallows the user to set filter criteria such as tag keywords and/orfilename or and/or subject, content or context specific words and othercriteria. The user may also choose to filter out tags and/or sub-tags byusing a feature that allows the user to mark the tag as shown (in thiscase with a ‘x’ sign 1270 as shown in exemplary embodiment). User cancreate multiple levels of tags and sub-tags as shown by 1272.

In the description above, a file categorizing system has been defined interms of tags that can be created and linked/associated with files andfolders. Users can view tags, as shown in FIG. 48B, instead of filenamesand folder names as in a standard file system. The tagging method canalso be used to tag websites while browsing. Tags can be used withdocuments, images, applications, and any other type of data. Files andfolders can be searched and the appropriate content retrieved by lookingup on one or a combination of tags associated with the files andfolders. Users may also simply specify tags and the AFMS would identifythe appropriate location to store/save/backup the file. In exemplaryembodiment, if a user is trying to save an image with the tag,‘Ireland’. The AFMS would identify the file as an image file and the tag‘Ireland’ as a place/destination that it identifies as not being in theuser's vicinity i.e., (not in the same city or country as the user).Then, the AFMS would proceed to store the file in an imagespace/section/file space in the subspace/subsection entitled or taggedas ‘My Places’ or ‘Travel’. If a subspace does not exist that alreadycontains pictures of Ireland, it would create a new folder with thename/tag ‘Ireland’ and save the image in the newly created subspace,else it would save the image to the existing folder containing picturesof ‘Ireland’. In another exemplary embodiment, the user may want to savea project file tagged as ‘Project X requirements’. The AFMS determinesthat there are associate accounts, as described later, that share filesrelated to Project X on the owner user's account. The AFMS proceeds tosave the file in the space tagged as ‘Project X’ and sets filepermissions allowing associate accounts that share Project X's space onthe owner user's account to access the newly saved file (Project Xrequirements). Thus, the AFMS/VOS not only determines the appropriateload/save location for files, but also the permissions to set for anynew file on the system. Additionally, the file and folder content may besearched to retrieve relevant files in a keyword search. Users may beprovided with the choice of viewing and searching for files according tothe standard mode i.e., file and folder names or they may opt for usingtagged content. This would offer greater control to users in terms ofvisualizing, managing and using their files and data. Data and filesthat are tagged provide the user with more flexibility in terms oforganizing and accessing data. In exemplary embodiment, a user may tag aphoto showing the user as a child with his mom on the beach, with theterm ‘childhood memories’. The user may tag the same photo with thephrase ‘My mommy and me’ and ‘beach’. Anytime the user searches for anyof the tags, the photo is included in the collection of photos (oralbum) with the given tag. Thus, a single photo can comprise multiplealbums if it is tagged with multiple keywords/phrases.

Applications are designed that make use of the tag concept. In exemplaryembodiment, one such application is a photo mixer/slideshow/displayprogram that takes as input a tag name(s), retrieves all photos with thespecified tag, and dynamically creates and displays the slideshow/photoalbum containing those photos.

Reference is now made to FIG. 49C. Applications 1280 may be provided bythe AFMS/VOS system. Alternatively, external applications may be addedto it. In the following figure, examples of two applications are shownin context in order to describe the immersive features of this system.The first application is a blog 1282. This application can beinstantiated (i.e., opens up) within the AFMS itself, in an exemplaryembodiment. If the blog exists on a website, then the user wouldnavigate to that site and edit its contents from within AFMS. Users canthen add multimedia content to their blog with ease. The AFMS providesan interface 1284 for viewing and using files that may be located eitheron the user's local machine or in the AFMS or on a remote machineconnected to the web. The file viewer/manager may open up in a sidebar1284 as shown in exemplary embodiment, or in a new dialog window or takesome other form which allows concurrent viewing of both the application1282 and files. Snapshots of files can be seen within this file manageras shown by 1284. The user can then simply drag and drop files for usein application 1282. Examples of this are shown in FIG. 49C. The usercan drag and drop images or videos 1286 for user with the blogapplication 1282. The following figure FIG. 49D shows the resultingeffect. Further, the complete file repository may be accessed by using anavigation scheme 1288 within the manager to view contents. Here acursor scheme 1288 is used to navigate within the file manager.

Reference is now made to FIG. 49D where the blog application 1282 isshown with the image and video files 1290 that were uploaded by draggingand dropping from their respective file locations using the file managerwindow 1284. The file manager window 1284 in FIG. 49D shows files thatinclude the tags ‘Products: HP’ and ‘Reviews: CNET’. Web links are shownsorted by date. The figure shows that hyperlinked content can also beembedded within applications via the file manager. Here the link isdragged and dropped 1292 demonstrating ease of use even in such cases.Reference is made to FIG. 49E where the result is shown. The hyperlinkedcontent appears with the title, source and a summary of the content. Theway this content appears can be modified by hovering with the mouse overthis content, in an exemplary embodiment. This causes a window 1296 toappear which shows options that the user can select to show/hide entirehyperlinked article content, or summary and/or the source of thecontent.

Reference is now made to FIGS. 49F-G where an example of immersive filefeatures comprising the AFMS/VOS is given with reference to anotherapplication. In this case, it is a notebook/scrapbook application 1300as shown in FIG. 49F. Options 1302 for customizing applications and/orchanging application settings will be present in the AFMS. Here too isshown the file manager window 1304 from which files under the relevanttags can be dragged and dropped 1306 to the appropriate location in theapplication 1300. FIG. 49G shows the results 1310 where the selectedmultimedia files have been uploaded to the application by a simple moveof the mouse from the file space to the application space within theAFMS. Content 1312 in the application may be edited or uploaded from thefile space right within the AFMS where the users have readily availabletheir file space, applications, the web and other resources.

Reference is now made to FIGS. 49H-L where the flexibility of filespaces and their content within the AFMS/VOS is described with referenceto an example. FIG. 49H presents the example at the top in terms of auser need. A user may want to create an exclusive file space (alsocalled ‘smart file spaces’) for books where they can store and manage avariety of file types and content. The AFMS/VOS allows the user tocreate such a section. The procedure starts off by creating and namingthe section and picking an icon for it which will be provided in acatalogue 1320 to users. Users may also add their own icons to thiscatalogue. The result is the user's very own book space 1326 which canbe referenced by the iconic section caption 1322. The user may decide toadd folders or tags in this space. One such tag/category is shown in1326 as: ‘Business decision-making’. As the user browses websites in thewebspace 1324 provided by the AFMS/VOS, the user can easily upload/copyappropriate content from the website/URL location into theircustom-built file section 1326. FIG. 49H shows the user dragging anddropping images of books that the user is interested in, into the bookssection 1326. The image content thus gets uploaded into the user'scustomized file space. Images and other content uploaded/copied from asite in this manner into a user's file space may be hyperlinked to thesource and/or be associated with other information relating to thesource. Users can add tags to describe the data uploaded into the filespace. The AFMS/VOS may automatically scan uploaded object for relevantkeywords that describe the object for tagging purposes. In the case ofimages, the system may use computer vision techniques to identifyobjects within the image and tag the image with appropriate keywords.This is equivalent to establishing correspondence between images andwords. This can be accomplished using probabilistic latent semanticanalysis [55]. This can also be done in the case of establishingcorrespondence between words (sentences, phonemes) and audio. FIG. 49Iillustrates that textual content/data may also be copied/uploaded intothe user's customized file space by selecting and copying the content inthe space. This content may be stored as a data file or it may be‘linked’ to other objects that the user drags the content over to, inthe file space. For instance, in FIG. 49I, the user drags the selectedcontent 1328 from the webspace 1324 over the image 1330. Hence thecopied content gets linked to this image object 1330. The linked contentmay be retrieved in a separate file or it may appear alongside theobject, or in a separate dialog or pop-up or window when the userselects the particular object, for instance, by clicking on it.

FIG. 49J shows the file space 1340 after content from the website hasbeen uploaded. The image objects 1342 along with their sourceinformation are present. The content 1344 (corresponding to the selectedtext 1328 in FIG. 49I) can be viewed alongside the linked image in thefile space 1340. Thus, the AFMS/VOS allows for creation and managementof ‘context specific file spaces’ where the user can easily load contentof different types and organize information that appears to go togetherbest, from a variety of sources and locations, in a flexible way, andwithout worrying about lower layer details.

Organization of information in these file spaces is not tied to datatype or file format or application being used, but instead all objectsthat appear to the user as belonging together can be tied together as asingle ‘information unit’. These information units can then organized asbigger information units and so on. In the current example the image ofthe book, its source and the content it is linked with together compriseone information unit. The book objects together (1342) comprise a higherinformation unit comprising the ‘My Books’ section. These informationunits stand apart from standard data files and folders because theycontain data of multiple types that is linked or associated together,and hence are flexible. Further, data and content of different typesfrom multiple sources can be assimilated together by the system whichwill handle the lower layer functionality to create these informationunits in a manner that is easy to access, view and manage, thusenhancing the value of the information to the user.

In FIG. 49J, additional examples are given to demonstrate other ways ofcombining data with information units. An object in a file space can becross-referenced with information or data from other applications thatis of relevance or related to that object. For instance, the book objector information unit 1346 can be cross referenced with web links, relatedemails and calendar entries as shown in 1348 and categorized usingrelevant tags. In this example, the user has added web links of storesthat sell the book, emails and calendar entries related to the subjectmatter and events involving the book. Thus, the user can easilyreference different types of files and objects that are related to thesame subject matter or object using the features of this file system.The information in any given smart file space can be used by theAFMS/VOS to answer user queries related to objects in the file spaces.For instance, in the present example, the user may query the AFMS forthe date of the ‘blink’ book signing event in the ‘My Books’ file space1340 in FIG. 49J. The AFMS identifies the ‘blink’ object 1346 in thefile space and looks up appropriate information linked to or associatedwith 1346. In this case since the query deals with ‘date’, the AFMSsearches for linked calendar entries and emails associated with 1346related to ‘book signing’, by parsing their subject, tags and content.In this case, the AFMS would identify and parse the email entry on booksigning in 1348 in FIG. 49J and answer the query with the relevant dateinformation.

In an exemplary embodiment of the smart file space implementation, eachfile space may be associated with an XML file. When an object or content(image, text, etc.) is dragged and dropped, the code underlying thecontent is parsed and the appropriate information and properties areidentified. This information includes type of content or data, contentsource, location, link information (for example, this is a link to animage of a house), content description/subject. Other information thatthe AFMS/VOS determines includes, the application needed to view or runthe object being saved into the file space. For instance, when an imageis dragged and dropped into a file space from a web page, the HTML codefor the web page is parsed by the AFMS in order to identify the objecttype (image) and its properties. Parsing the image source tag (<src>) inthe HTML file for the web page provides the source information for theimage, in exemplary embodiment.

In FIG. 49K, collaborative features of the AFMS/VOS and its associatedfile management features are described. Users can maintain a list offriends 1260 and their information in the AFMS/VOS. These friends canhave limited access accounts on this system (called ‘associate’ accountsdescribed later) so that they can access and share the primary user'sresources or interact with the primary user. Users can set options toshare information units/objects in their file spaces, such as bookobject 1362 in the ‘My Books’ section 1326 in FIG. 49K, with theirfriends. Users can drag and drop objects directly onto a friend'simage/name in order to share those objects with the friend. Anotherfeature in this file system is that when an object 1362 in the filespace 1326 and friends 1364 from the friends list 1360 are selectedconcurrently, a special options window 1366 pops up that presentsfeatures relevant to the ‘sharing’ scenario. The AFMS/VOS recognizesthat selections from both the friends list and file space have been madeand presents users with options/features 1366 that are activated onlywhen such a simultaneous selection occurs and not when either friends orfile space objects are exclusively selected. Some of these options areshown in 1366 in exemplary embodiment. For instance, users can set grouptasks for themselves and their friends involving the selected object,such as attending the author signing event for the book 1362. Otheroptions include turning on updates, such as the addition of objects, fora section to the selected friends; going on a shopping trip for theobject with selected friends.

Owners may be able to keep track of physical items they lend to orborrow from their friends. An object in a file space may be a virtualrepresentation of the physical item. Users can set due dates orreminders on items so that items borrowed or lent can be tracked andreturned on time. A timestamp may be associated with a borrowed item toindicate the duration for which the item has been borrowed. Thismethod(s) to keep track of items can serve as a Contract ManagementSystem. This service can be used to set up contracts (and other legaldocuments) between users using timestamps, reminders and other featuresas described. Witnesses and members bound to a contract may establishtheir presence during contract formation and attestation via a webcam orlive video transmission and/or other electronic means for live videocapture and transmission. Members bound to a contract and witnesses mayattest documents digitally (i.e., use digital signatures captured byelectronic handwriting capture devices for example). Users may alsocreate their WILL through this system. User authenticity may beestablished based on unique pieces of identification such as theirSocial Insurance Number (SIN), driver's license, passport, electronicbirth certificate, retinal scans, fingerprints, health cards, etc.and/or any combination or the above. Once the authenticity of the userhas been verified by the system, the system registers the user as anauthentic user. Lawyers and witnesses with established credibility andauthenticity on the system may be sought by other users of the systemwho are seeking a lawyer or witness for a legal documentsigning/creation for example. The credibility of lawyers, witnesses andother people involved in authenticating/witnessing/creating a legaldocument may further be established by users who have made use of theirservices. Based on their reliability and service, users may rate them inorder to increase their credibility/reliability score through thesystem. Thus, group options involving data objects and users is a uniquefile management feature of the AFMS/VOS that allows for sharedactivities and takes electronic collaboration to a higher level. TheContract Management System may be used/distributed as a standalonesystem.

FIG. 49K shows options/features 1370 that are presented for managing aninformation unit upon selecting the particular object or informationunit 1368 in a file space. These options allow users to send an email orset tasks/reminders related to the object; tag the object, link otherobjects; receive news feeds related to that object; add it to anotherfile space; and perform other tasks as given in 1370.

In another exemplary embodiment of AFMS/VOS usage for informationlookup, a user may want to look up information on the last clientmeeting for a specific project. The file space for the project, createdby the user, would contain the calendar entry for the last meeting, theemail link containing the meeting minutes as an attachment, and otherrelated objects and files. The user may also share the project filespace with other users involved in the project by adding them as‘friends’ and sharing the file space content, in exemplary embodiment.Thus, the smart file space saves the user time and effort as the user nolonger has to perform tedious tasks in order to consolidate items thatmay ‘belong together’ according to a user's specific needs. Forinstance, in this case the user does not need to save the meetingminutes or the email content separately; just dragging and dropping theappropriate email from the email application to the project's file spacesuffices and the email and attachment are automatically linkedto/associated with the project. The user does not have to open thecalendar application and tediously browse for the last calendar entrypertaining to the meeting. Also, sharing the project space withcolleagues is easy so that project members can keep track of all filesand information related to a project without worrying about who has ordoesn't have a particular file. Other information may be available tousers sharing a file space such as the date and time a particular filewas accessed by a user, comments posted by shared users etc.Additionally, tools to ease file sharing and collaboration may beavailable via the VOS as described below with reference to FIG. 20.

FIG. 49L represents an exemplary embodiment of the storage structure ofthe AFMS/VOS. Data stored on a user's local machine or remote sites orservers such as a user's work machine, or online storage, and data ofuser's friends on the system is managed by the file management layer.The file management layer handles conflict analysis, filesynchronization, tagging, indexing, searching, version control, backups,virus scanning and removal, security and fault protection and otheradministrative tasks. Data (modification, updates, creation, backup) inall user and shared accounts on local or remote machines, on webservers, web sites, mobile device storage and other places can besynchronized by this layer. A property of the file system is that itcaches files/and other user data locally when network resources arelimited or unavailable and synchronizes data as network resources becomeavailable, to ensure smooth operation even during network disruptions.Backups of data conducted by AFMS may be on distributed machines. Anabstract layer operates on top of the file management system andprovides a unified framework for access by abstracting out the lowerlayers. The advantage of this is that the VOS offers locationtransparency to the user. The user may log in anywhere and see aconsistent organization of files via the VOS interface, independent ofwhere the files/data may be located or where the user may be accessingthem. The VOS allows users to search for data across all of the user'sresources independent of the location of the data. Another feature ofthis file system is the option of storing a user's information, profileand other account resources on the user's resources (for example, theuser's home or work computer) instead of a web server to ensure privacyof a user's data and account information. FIG. 49P demonstrates anexemplary embodiment of an application execution protocol run by theApplication Resource Manager ARM (which is a part of the virtualoperating system). Once a user requests an application 1400, the ARMchecks to see whether this application is available on the portal server1402. If so, then the application is run from the portal server 1404. Ifnot, then the application plug-in is sought 1406. If the plug-in exists,the application is run from the local machine 1412. If a plug-in for theapplication does not exist, a check for the application on the localmachine is conducted 1410. If available, the application is executedfrom the client's local machine 1412. If not, the application is runfrom a remote server on which the user has been authenticated (i.e., hasaccess permission) 1414, 1416. If all the decision steps in thealgorithm in FIG. 49P yield a negative response, the ARM suggestsinstallation paths and alternate sources for the application to the user1418. The user's data generated from running the application is savedusing the distributed storage model.

Another feature of the AFMS is that the user may store files in a“redirect” folder i.e., files moved/saved to this folder are redirectedby the AFMS to the appropriate destination folder based on the file'stags and/or content. The user may then be notified of where the file hasbeen stored (i.e., destination folder) via a note or comment or link inthe “redirect” folder that directs the user to the appropriatedestination. An index file may automatically be generated for foldersbased on titles/keywords/tags in the documents and/or the filename. Thisindex may display titles/keywords/tags along with snapshots of thecorresponding files.

Reference is now made to FIG. 49M where a user accounts managementstructure is shown. Central to this system is a user management layerthat manages a given ‘owner’ user's accounts as well as ‘associate’accounts, which would include accounts of all other friends, users andgroups (the owner would like to associate with). Associate accountswould be created to give access to the owner account resources and data.The owner account would have all administrative rights and privileges(read, write, execute, for example) and can set permissions on associateaccounts to grant or restrict access to the owner's account andresources. An associate account may be viewed as ‘the set of all ownerresources that the associate user has access to, and the set of allactivities that the associate user can engage in with the owner user’.An associate account would be linked to and accessible from theassociate user's primary/owner account. The owner account may beaccessible to and from the owner user's computer, to and from a machineat remote locations such as the office, to and from accounts at socialnetworking sites, and through a web browser/web sites. Accountinformation such as usernames and passwords for the user's accounts onwebsites and other servers that the user accesses from the VOS may bestored on the system so that the user bypasses the need to enter thisinformation every time the user accesses their external account. Theowner may set group policies for the associate accounts so that theyhave access to specific resources and applications for specific timeperiods on the owner's account. Owner users have the option ofclassifying associate users into categories such as acquaintances fromwork, school, family, strangers etc. As described before, users mayshare information and files with their friends, and also engage inshared activities such as games, edit documents collaboratively etc.,through the VOS. Another feature of the VOS is that over time it allowsthe user to specify automatic changes in access privileges/permissionsof associate accounts on the user's network. In exemplary embodiment, auser may want to let associates accounts, starting out with limitedaccess/privileges, have access to more resources over time. Through theVOS, the user is able to specify the resources that associate accountsmay automatically access after a certain period of time has elapsedsince their account was created or since their access privileges werelast changed. The user may also be able to grant greater accessprivileges automatically to associate accounts after they demonstrate acertain level of activity. After the specified period of time elapses orthe level of activity of an associate account increases/decreases or ismaintained, the VOS automatically changes the access privileges of theassociate users who have been granted access to increased/decreasedresources as pre-specified by the user through options provided by theVOS. This is the ‘Growing Relations’ feature of the VOS where accessprivileges rules of associate accounts are specified by a user and arechanged accordingly by the system, as and when specified by the user.The VOS is able to regulate resource use and change access privilegesautomatically in the absence of user specified access privilege rules,in another exemplary embodiment. The VOS may monitor activity levels ofassociate accounts and interactivity between user and associate usersand automatically determine which associate users may be allowed greateraccess privileges. If there is greater interactivity over time betweenthe user and a certain associate user, then the system may deem thisassociate user as a ‘trusted associate’. It may also use other means ofdetermining the ‘trustworthiness’ of an associate user. The system mayseek permission of the user before changing access privileges of thetrusted associate user. As the ‘trust score’ (the method used by thesystem to keep track of the activity levels of an associate account) ofan associate user increases, the system would promote the status of theassociate account progressively by assigning status levels such as:Stranger, Acquaintance, Friend, Family—in that order from first to last.The higher the status of an account, the more access privileges aregranted to that account. In a similar manner, if the VOS detects thatthere is little interactivity of an associate account over time, ordetermines lower resource needs of an associate account or assesses thatan associate account is less ‘trustworthy’ based on usage patterns ofassociate account users, then the VOS would regress the status of theaccount and grant less privileges accordingly. The system may again seekthe permission of the user before modifying access privileges of anyassociate account.

The VOS allows password synchronization across websites, networks andmachines. For example, if a user changes a password for logging onto alocal machine, say a home computer, the password change is synchronizedwith a password the user may use to login to their account on a webpage.Various levels of access privileges may be granted by the VOS to users,including but not limited to that of a root user, administrator, regularuser, super user, guest, limited user, etc., in exemplary embodiment.The VOS also allows execution of shell commands. VOS also provides asoftware development kit for users to write applications for the VOS.

The system may also contain an immersive search engine application thatperforms searches on queries presented to it. The search engine may beavailable as a standalone feature for use with browsers and/or networkmachine(s) or local machine browsing applications. It may be availableas part of a VOS browser, containing one or more of the VOS's features.Some of the features unique to this immersive search engine aredescribed next. Reference is made to FIG. 49N where abstraction of asearch query is demonstrated in exemplary embodiment. When a userperforms a search, the input is not limited to typing text and using akeyboard. Instead a new approach is proposed, where the input could bespeech to text, or mouse gestures or other data. In another example, auser may be able to drag and drop content from a newsfeed into thesearch query field. Context level searches may be performed by thesearch engine. In an exemplary embodiment, when a user comes across animage while browsing the web, the user may be able to simply drag anddrop the image into the search field and the browser would retrievesearch results that pertain to the image objects, theme or subject. Theuser may quote a sentence and the search engine would retrieve searchesrelated to the underpinning of that statement in a context relatedsearch, in another exemplary embodiment. Thus, this method effectivelyprovides a layer of abstraction for the conventional search. The searchengine can also retrieve search results in the form of lists where eachlists contains the results that fall under a specific category orcontext. Categories and sort criteria may be user specified. In anexemplary embodiment, the user may want to search for cars of aparticular year and want them categorized according to color, mostselling, safety rating and other criteria. The search engine thenretrieves search results of cars of that year sorted according to thespecified criteria in different lists. It also keeps track of userinformation so that it can provide contextual information specific orrelevant to the user's life. For example, if a user's friend has a carwith the specifications that the user is searching for, then the searchengine indicates to the user that the user's friend has a car with thesame or similar specifications. The search engine mines the informationunits present in a user's directory in order to present relevantcontextual information along with search results. For instance, the usermay be interested in six cylinder engine cars as inferred by the systembased on information objects in the user's directory. The search enginethen indicates to the user as to which of the search results pertain tosix cylinder engine cars. This type of contextual data mining can bedone as discussed in reference to FIG. 6E. Additionally, this searchengine can present to the user information in a variety of formats, notnecessarily restricting the search output to text. For instance, theresults may be converted from text to speech.

Users can save sites and bookmark sites using tags while browsing webpages. In exemplary embodiment, this can be done using the VOS browseror any other browser that supports tagging and saving. The tags can thenbe used by web crawlers to rank pages for use in search engines.Conventionally, web crawlers used by search engines rely primarily onthe keywords provided by authors of websites, as well as content on webpages. The method described here also utilizes tags provided by ordinaryusers browsing websites. This method also allows sites to be searchedwhich are not registered with the search engine.

Reference is now made to FIG. 49 O where an exemplary embodiment of theVOS is shown running as a website. The user may be presented with thisscreen upon logging in. There are many applications available for use inthe VOS. An API is also available for developers to build applicationsfor the VOS. Any of the applications such as text editors, spreadsheetapplications, multimedia applications (audio/video, photo and imageediting), white board can be used collaboratively with other usersthrough an intuitive interface. Collaborative application sharing may beaccomplished using techniques discussed with reference to FIG. 7A, B, C,D. Shared users may include friends/family members/other associates fromsocial networking sites or work or home computer accounts. Any changesmade to data or applications and other resources can be viewed by allusers engaged in the collaboration of these resources and accounts.Users can customize i.e., change the look and feel of the VOS includingthe background, theme etc. The VOS may also provide an interface thatallows for text, video and audio overlay. The calendar feature in FIG.49 O cross-checks calendars of all users for scheduling an event or anappointment or a meeting and suggests dates convenient for all usersinvolved. A time-stamping feature is also available that lets userstimestamp documents. This feature also has an encryption option thatallows users to encrypt documents before uploading, acquire a timestampfor the document and retrieve it for future use, keeping theconfidential all the while. This might serve useful where time-stampingdocuments serves as proof of ownership of an invention, for example.Encryption may be accomplished using two encryption keys in exemplaryembodiment. One key would be available only to the user and the systemwould maintain the other key. Remote technical assistance is alsoprovided for this interface. FIG. 49 O also incorporates advanced search(described previously with reference to FIG. 49N), distributed dataaccess (FIG. 49L), advanced user management (FIG. 49M), safety depositbox, media room, launch pad, library, TV/radio and other features asshown in FIG. 49 O. The ‘safety deposit box’ would contain sensitivematerials such as medical records, legal documents, etc. These contentsare encrypted and password protected. In an exemplary embodiment, datais encrypted at the source before backing it up on machines. The filesmay also be accessible or linked to commercial and other public orprivate large-scale repositories. For instance, medical records of auser could be linked to or accessible from hospital repositories towhich a user has been granted limited access. Application layers may beadded that mine the contents of the safety deposit box in order topresent information to the user in functional and relevant manner. Inexemplary embodiment, a ‘calendar alert’ application may remind the userof pending actions. For instance, based on their medical record, theapplication would alert the user that a vaccination is due, or a dentistappointment is due. In another instance, the application would alert theuser based on financial records that their taxes are due. Similarscenarios may exist for legal documents. The ‘media room’ would includeall files and folders and content that the user wishes to publish ormake public such as web pages, videos (such as YouTube videos) etc. Thelaunch pad is a feature that allows users to place objects in a regionand take appropriate actions with those objects. It provides aninterface for programming actions that can be taken with respect toobjects in a variety of formats. The launch pad includes community userswho can contribute their applications and other software for use. Inexemplary embodiment, a user may move 2D onto a “3D-fy” applicationwidget in the launch pad section in order to transform the 2D imagesinto their corresponding 3D versions. In another exemplary embodiment, auser may add an application in the launch pad area that allows documentsharing and editing through a webcam. The library section may includee-documents such as e-books, electronic articles, papers, journals,magazines etc. This section will be equipped with the facility wherebyelectronic magazines, e-papers etc. to which the user may havesubscriptions would be ‘delivered’ by the subscribed service and madeavailable in this section. The TV/radio feature allows users to browseand view channels in a more traditional sense online. The channels maybe browsed using the keyboard or mouse. It may also be combined with theuser interface discussed with reference to FIG. 54D. The output of cableTV could also be viewed via this facility. In exemplary embodiment, thiscan be done by redirecting output from the user's TV or cable source tothe user's current machine via the internet or network. The channels canbe changed remotely, for example via the interface provided by the VOSor a web interface independent of the VOS. In exemplary embodiment, thismay be done by connecting a universal TV/radio/cable remote to a homecomputer and pointing the device towards the object being controlled viathe remote, if necessary (if it's an infrared or other line-of-sightcommunication device). A software on the computer communicates with theremote to allow changing of channels and other controls. The audio/video(A/V) output of the TV or cable is connected to the computer. Thecomputer then communicates with the remote device over the Internet, fordisplay/control purposes in exemplary embodiment. The TV/radio contentmay include files, and other media content on the user's local or remotemachine(s), and/or other user accounts and/or shared resources. Theradio may play live content from real radio stations. The system mayalso allow recording of TV/radio shows. On logging off the VOS, thestate of the VOS including any open applications may be saved to allowthe user to continue from where the user left upon logging in again. Anyactive sessions may also persist, if desired.

FIG. 49Q provides an additional exemplary embodiment of file tagging,sharing and searching features in the VOS/AFMS. As a user browses webpages in a web browser 1440, which may be the VOS browser, the user maychoose to save web page content such as an image 1442. The user would beable to choose the format to save it in, and also edit and savedifferent versions of the image. Here the image 1444 is shown with aborder around it. The user can tag images to be saved using keywords1446. Parts of the image can also be labeled as 1448. The user canspecify friends and associate users to share the image with 1450. Thelocation 1454 of the image/file can be specified in abstract terms. Forinstance, the user can specify the location where the file is saved suchas the home or office machine, or ‘mom's computer’. Owing to thedistributed file storage nature of the VOS, the lower layers can beabstracted out if the user chooses to hide them. The VOS is based on alanguage processing algorithm. It can recognize keywords and sort themaccording to grammatical categories such as nouns, verbs, adjectivesetc, by looking up a dictionary in exemplary embodiment. It can learnthe characteristics of the associated word based on the image. Morespecifically, the user may be able to train the algorithm by selecting akeyword and highlighting an object or section of the image to create theassociation between the keyword and its description. For instance, theuser may select the keyword ‘horse’ and draw a box around the horse inthe image, or the user may select ‘white’ and click on a white area inthe image. In this way, the system can be ‘contextually’ trained.Similar training and associative learning can occur in the case of audioand video content. Based on the image keywords, labels and associatedcharacteristics learnt, the system would be able to make contextualsuggestions to the user. In exemplary embodiment, the user may searchfor a ‘black leather purse’. The VOS would remember search terms for aperiod of time and make suggestions. So for instance, if an associateuser or someone on the user's friend list bought a leather purse, thesystem would notify the user of this fact and the source/store/brand ofthe purse and check the store catalogue from which the purse was bought,for similar or different purse in ‘black’ and/or ‘leather’. In anotherexemplary embodiment, the system would inform a user ‘A’ of photos thatan associate user ‘B’ has added containing user A's friend whom the userA wishes to receive updates on. The VOS presents search results in a‘user-friendly’ manner to the user. Some aspects may be pre-programmed,some aspects may be learned over time by the VOS with regards whatconstitutes a user-friendly presentation, whether it involves displayingimages, videos, audio, text, and any other file or data in any otherformat to the user. In exemplary embodiment, a user may search for afriend's photos and the VOS would display images found of the user'sfriend after properly orienting them, by applying affine/perspectivetransformations for example, before displaying them to the user. Theuser's friend may also be highlighted by using markings or by zoomingin, as examples in order to make it easier for the user to identifytheir friend in a group, for instance. User may conduct search usingfilters or terms that are adjectives such as ‘dark’, ‘purple’, ‘thick’,‘lonely’ etc., as well as any class of words that can be used todescribe or characterize a noun/object. The VOS searches for relevantinformation matching these search terms/filters based on tags associatedwith files and objects. Additionally, computer vision techniques can beused to characterize whole images/video sequences, and objects andcomponents within images/videos.

If the user is listening to a soundtrack, the system can make comments,based on user's mined data, such as ‘it's your friend's favourite musictrack’. It can analyze the soundtrack and find tunes/music similar tothe one the user is listening to. It can identify other soundtracks thathave been remixed by other users with the track the user is listening toor find soundtracks compatible with the user's taste etc. Extraction offamiliar content can be done by the system in exemplary embodiment usinga mixture of Gaussians [56] or techniques similar to those in [57]. Theuser would be able to specify subjective criteria and ask the system toplay music accordingly. In exemplary embodiment, the user can specifythe mood of the music to listen to, for instance—sad, happy,melodramatic, comical, soothing, etc. Mood recognition of music can beperformed via techniques specified in [58]. The system can also monitoruser activities or judge user mood through a video or image capturedevice such as a webcam and play music accordingly or make comments suchas ‘hey, you seem a little down today’ and play happy music or suggestan activity that would make the user happy or show links that arecompatible with the user's interests to cheer the user up. The trackscan be played either from the user's local machine or from online storesand other repositories or from the user's friends' shared resources.Detecting the mood underlying a soundtrack and content similar to asoundtrack can be detected using techniques specified in [59].

The VOS can make recommendations to users in other areas byincorporating user preferences and combining them with friend'spreferences, as in the case of a group decision or consult i.e.,‘collaborative decision-making or consulting’. In exemplary embodiment,users may specify their movie preferences such as ‘action, ‘thriller’,‘drama’, ‘science fiction’, ‘real life’, etc. They may specify othercriteria such as day and time of day they prefer to watch a movie,preferred ticket price range, preferred theatre location, etc. In anonline collaborative environment, such as that shown in FIG. 20 inexemplary embodiment, users may consult with each other or plantogether. For example, a group of friends may want to go and watch amovie together. Every user has their own movie preference, which thesystem may incorporate to suggest the best option and other associatedinformation, in this case the movie name, genre, show time etc. Othertools and features to facilitate group decisions include taking votesand polls in favour or against the various options available to theusers. The system would then tally the votes and give theanswer/option/decision that received the maximum votes. The system mayalso incorporate general information about the subject of decision inorder to make recommendations. For instance, in the movie example, thesystem may take into account the popularity of a movie in theatres(using box office information for example), ticket deals for a movie,etc. in order to make recommendations. Users can also use the modes ofoperation described with reference to FIG. 7 for collaborativeapplications on the VOS. For example, when editing a filecollaboratively, as a user edits a document, he/she can see theadditions/modifications that are being made by other users.

Reference is made to FIG. 49R where an example of a user interface forfiltering search data is shown. Users can filter files on the basis oflocation, file types or by file author(s).

Reference is now made to FIG. 49S where an exemplary embodiment of anobject oriented file system is shown. Users can specify the structure ofa folder (used for storing files on a computer). For example, as shownin the figure, a user can create of folder of type “company” in whichthe user specifies a structure by creating entries for subfolders oftype “HR”, “R&D”, “Legal”, and “IT”. Regular folders may also becreated. Each of the created folders can have its own structure. Theuser can have a folder listing all the folders of type “company” asshown in the box on the left in the top row of FIG. 49S. The content ofa selected folder is shown in a box on the right in the top row. Theuser has options to view by “company” or by the structures thatconstitute that folder, say by “HR”. In FIG. 49S, the top row shows anexample of viewing by “company”. If the user chooses to view by “HR”,the view on the right (as shown in the bottom row of FIG. 49S) displaysthe all the HR folders organized by “company”. Other filters are alsoavailable to the users that search according to the desired fields of afolder. Arrows are available on the right and left of the views to gohigher up or deeper into folders. In another exemplary embodiment,instead of having a structure, the folders and files can have tags thatdescribe the folder and the files. The proposed object oriented filesystem simplifies browsing and proves the advantages of a traditionalfile system and a fully fledged database.

Reference is now made to FIG. 20. The collaborative interface shown inFIG. 20 for a shopping trip may be used in the case of othercollaborative activities such as application, file, document and datasharing. A generic version of the interface FIG. 20 is now described inexemplary embodiment to illustrate this extension. Panel 241 listsfriends involved in the collaboration. An application panel replaces thestore panel 242 and displays shared applications of users involved inthe collaboration. Panel 247 lists the user's documents, data files andother resources. Panel 248 lists the user's friends' documents, datafiles and other resources. Window 243 would facilitate collaborativesharing of applications, documents, data, other files and resourcesbetween users of a collaboration. Users can direct any signal to243—video, audio, speech, text, image, including screen capture, i.e.,they may specify a region of the screen that they wish to share in 243,which could include the entire desktop screen. (A perspective correctionmay be applied to documents that are being shared. For example, if avideo of a talk is being shared and the video of the slides of thepresentation is being shot from an angle (as opposed to the camera beingorthogonal to the screen), a perspective transform may be applied sothat lines of text on the screen appear horizontal to ease viewing)Users may be able to drag and drop applications, files, documents, data,or screenshots as well as contents/files captured by the screenshots andother resources into window 243 during collaborative sharing. Instancesof collaboration include shared use of applications; viewing, creating,editing, saving documents or image files etc. Window 243 has a visualoverlay for users to write or draw over to permit increasedinteractivity during collaborative discussions. This is analogous towhiteboard discussions except that here the overlay may be transparentto permit writing, scribbling, markings, highlighting over content beingshared in 243. All this content may be undone or reversed. The overlayinformation can be saved without affecting the original content in 243if the user chooses to do so. Overlay information can be saved inassociation with the original content. The system also allows a ‘snap toobject’ feature which allows users to select and modify objects in theview. The toolbar 239 provides overlay tools and application and/ordocument and file specific tools for use with the specific applicationand/or file or document or data being shared in 243. View 243 alsosupports multiple layers of content. These layers could be hidden orviewed. The screen size of 243 is resizable, movable, dockable,undockable. All sessions and content (viewed, edited, text, speech,image, video, etc.), including collaborative content and information maybe saved including all environmental variables. When editing a filecollaboratively, as a user edits a document, he/she can see theadditions/modifications that are being made by other users.Collaborative environments such as these can be specialized to cater tooccupation, age group, hobby, tasks, and similar criteria. In anexemplary embodiment, a shared environment with features described abovemay exist for students where they can collaborate on homeworkassignments and group projects as well as extracurricular activitiessuch as student council meetings, organization of school events etc.Specialized tools to assist students collaborate on school relatedactivities is provided with toolbar 239. This environment would alsocontain applications specific to the context. For instance, in thestudents' collaborative environment, students would be able to providereviews on courses or teachers using the application provided for thispurpose.

Furthermore, the whiteboard may be integrated with a ‘convert tophysical model’ feature that transforms a sketch or other illustrationor animation on the whiteboard to an accurate physical model animationor video sequence. This may be accomplished via techniques similar tothose described in [3]. In exemplary embodiment, a user may draw a ballrolling on a floor which then falls off a ledge. The physics feature mayconvert the sketch to an animation sequence where the floor has afriction coefficient, and the ball follows Newton's Laws of Motion andthe Laws of Gravitation while rolling on the floor or free-falling. Inaddition, voice to model conversion may occur where the semanticsunderlying speech is analyzed and used to convert to a physical model.This may be accomplished by converting speech to text and then text topicture [60] and then going from picture to model [3]. Objects seen in awebcam may be converted to a model [3]. Users can then be allowed tomanipulate this object virtually. The virtual object's behaviour may bemodeled to be physically plausible. Based on the content of thewhiteboard deciphered through OCR optical character recognitiontechniques or sketch to model recognition [3] or speech to modelrecognition, related content (for example advertisements) may be placedin the proximity of the drawing screen and/or related content may becommunicated via audio/speech, and/or graphics/images/videos.

The interface shown in FIG. 20 may be used for exhibitions, wheredifferent vendors can show their product offerings.

Reference is now made to FIG. 51A where a communication networkdemonstrating external connections to system 10, is shown in exemplaryembodiment. FIG. 51A shows devices, systems and networks that system 10can be connected to, in exemplary embodiment. System 10 is connected tothe Public Switched Telephone Network (PSTN), to cellular networks suchas the Global System for Mobile Communications (GSM) and/or CDMAnetworks, WiFi networks. The figure also shows connections of system 10to exemplary embodiments of computing applications 16, and exemplaryembodiments of computing devices 14, such as a home computing device, awork computing device, a mobile communication device which could includea cell phone, a handheld device or a car phone as examples. The AFMS/VOSmay be connected to external devices, systems and networks in a similarmanner as system 10. The AFMS may additionally be connected to system 10itself to facilitate shopping, entertainment, and other services andfeatures available through system 10.

In the following discussion, a ‘Human Responder Service’, itsfunctionality and application is described. This service makes use ofthe data, and applications connected to the network shown in FIG. 51A.This service may be available on the portal server 20 as part of system10, or it may be implemented as part of the virtual operating system, orit may be available as an application on a home server or any of thecomputing devices shown in FIG. 51A and/or as a wearable device and/oras a mobile device. The Human Responder Service or Virtual Secretary isa system that can respond to queries posed by the user regarding userdata, applications or services. The system mines user data andapplication data, as well as information on the Internet in order toanswer a given query. An exemplary embodiment of a query that a user canpose to the system through a mobile communication device (cell phone orhandheld in an exemplary embodiment) includes “What is the time andlocation of the meeting with Steve?” or “What is the shortest route tothe mall at Eglinton and Crawford road?” or “Where is the nearest coffeeshop.” Further refinements in the search can be made by specifyingfilters. An exemplary embodiment of such a filter includes a time filterin which the period restriction for the query may be specified such as“limit search to this week” or “limit search to this month”. The filtersmay also be as generic as the query and may not necessarily berestricted to time periods. The input query may be specified in text,voice/audio, image and graphics and/or other formats. In an exemplaryembodiment, the user can send a query SMS via their mobile device to theVirtual Secretary (VS) inquiring about the location of the party theuser is attending that evening. On receiving the SMS request, the VSlooks up the requested information on social networking sites such asFacebook of which the user is a member, the user's calendar and email.After determining the requested information, the VS then responds to theuser by sending a reply SMS with the appropriate answer. If multiplepieces of information are found, the VS may ask the user which piece ofinformation the user would like to acquire further details on. The usermay also dictate notes or reminders to the VS, which it may write downor post on animated sticky notes for the user.

In an exemplary embodiment, the VS may be implemented as an application16 on a home computing device 14 that is also connected to the homephone line. Calls by the VS can be made or received through VoIP(Voice-over-Internet-Protocol) or the home phone line. The VS can alsobe connected to appliances, security monitoring units, cameras, GPS(Global Positioning Systems) units. This allows the user to ask the VSquestions such as “Is Bob home?” or “Who's at home?” The VS can monitorthe activity of kids in the house and keep an eye out for anomalies asdescribed with reference to FIG. 52B. Prior belief on the location ofthe kids can come from their schedules which may be updated at any time.Other services are available to the user include picking up the homephone and asking the VS to dial a contact's number, which the VS wouldlook up in the user's address book on the user's home computer or on asocial networking site or any of the resources available through theVOS. The user may click on an image of a user and ask the VS to dial thenumber of that user. The user may point to a friend through a webcamconnected to the VS and ask the VS to bring up a particular file relatedto the friend or query the VS for a piece of information related to thefriend. The VS may also monitor the local weather for anomalies, andother issues and matters of concern or interest to the user. Forinstance, if a user is outside and the VS system is aware of a snowstormapproaching, it sends a warning notification to the user on their mobilephone such as, “There is a snow-storm warning in the area John. It wouldbe best if you return home soon.” Other issues that the VS may monitorinclude currency rates, gas prices, sales at stores etc. Thisinformation may be available to or acquired by the VS via feeds from theinformation sources or via websites that dynamically update the requiredinformation.

One exemplary embodiment of the VS is described in FIG. 51B. The systemwaits for external user commands. Commands can come via SMS,voice/audio/speech through a phone call, video, images. These commandsare first pre-processed to form instructions. This can be accomplishedusing speech-to-text conversion for SMS, voice/audio/speech; parsinggestures in videos; and processing images using methods described withreference to FIG. 52A. These instructions are then buffered into memory.The system polls memory to see if an instruction is available. If aninstruction is available, the system fetches the instruction, decodesand executes it, and sends it back to memory. The response in memory isthen preprocessed and communicated to the external world.

The VS answers queries by looking up local information—user, applicationdata on the local machine, and then proceeds to look up information inother networks to which the user has access, such as web-based socialnetworks, and the internet. It may also automatically mine and presentinformation where applicable. In exemplary embodiment, when a userreceives a phone call, the VS searches address books on the localmachine and/or the Internet and/or social networks such as web-based,home or office networks to look up a person's name, phone and otherinformation, including pictures, and display the appropriate informationduring an incoming phone call. If the information is not on any of theuser's networks, the VS may look up public directories and other publicinformation to identify caller and source. The VS may also lookup/search cached information that was previously looked up or that isavailable on the local machine. Additionally, the VS gives informationabout the type of caller and relation between caller and user. Forinstance, the VS informs the user whether the call is from atelemarketing agency or from the dentist or from Aunt May in SanFrancisco etc. The VS may also specify location of the caller at thetime of the call using GPS and positioning and location techniques. TheVS may make use of the colloquial language to communicate with the user.The call display feature can be used as a standalone feature with cellphones and landlines and VoIP phones. A user may query the VS with ageneric query such as ‘What is an Oscilloscope?’ The VS conducts asemantic analysis to determine the nature of the query. In this case, itdetermines that the query is related to a definition for a term. In thiscase, it would look up a source for definitions such as anencyclopaedia, based on its popularity and reliability as a source ofinformation on the internet, or as specified by the user. As an example,it may look up Wikipedia to answer the user's query in this case.

Services based on identifying users' location are available through theVS. The VS may also be linked to, accessible to/by mobile phones orhandheld devices of members in the user's friends' network, businessesin the user's network and other users and institutions. Location can becomputed/determined using mobile position location technologies such asthe GPS (Global Positioning System) or triangulation data of basestations, or a built in GPS unit on a cell phone in exemplaryembodiment. The VS can inform the user if friends of the users are inthe location or vicinity in which the user is located at present; and/orindicate the position of the user's friend relative to the user and/orthe precise location of a given friend. In exemplary embodiment, if theuser is at a grocery store, and the VS detects that a friend (George) ofthe user is around the corner, then the VS may point this out to theuser saying, “Hey, George is at the baked goods aisle in the store.” Inorder to establish location in the store, the VS may establish acorrespondence between the GPS location coordinates on the store mapavailable via the retail server 24. The VS may additionally overlay thelocation coordinates on a map of the store and display the informationon the user's handheld device. The VS may display a ‘GPS trail’ thathighlights the location of a user over time (GPS location coordinates inthe recent past of a user being tracked). The trail may be designed toreflect age of data. For example, the colour of a trail may vary fromdark to light red where the darker the colour, the more recent the data.The users may communicate via voice and/or text and/or video, and/or anycombination of the above. The content of the conversation may bedisplayed in chat boxes and/or other displays and/or graphics overlaidon the respective positions of the users on the map. Also, the user canquery the VS to identify the current geographic location of a friend atany given time. Therefore, identification of a friend's location is notnecessarily restricted to when a friend is in the user's vicinity. Usersmay watch live video content of their friend on their mobile device fromtheir location. They may interact with each other via an overlaidwhiteboard display and its accompanying collaborative tools as describedwith reference to FIG. 20. In exemplary embodiment, with reference toFIG. 56, ‘User A’ may be lost and he may phone his friend, ‘User B’ whocan recognize the current location of User A based on the landmarks andvideo information User A transmits via his mobile. User B may alsoreceive GPS coordinates on her mobile via the VS. User B can thenprovide directions to User A to go left or right based on the visualinformation (images/video) that is transmitted to User B's mobile viaUser A's mobile. User B may also scribble arrows on the transparentoverlay on the video, to show directions with reference to User A'slocation in the video, which would be viewable by User A. Based on thecontent of the whiteboard deciphered through OCR optical characterrecognition techniques or sketch to model recognition [3] or speech tomodel recognition, related content (for example advertisements) may beplaced in the proximity of the drawing screen or anywhere else on themobile's screen and/or related content may be communicated viaaudio/speech, and/or graphics/images/videos.

Users may request to appear invisible or visible to friends and/or otherusers, so that they cannot be located by the user they appear invisibleto. Businesses may use location data for delivery purposes in exemplaryembodiment. For instance, pizza stores may deliver an order made via theVS to the user placing the order, based on their GPS coordinates. Userscan request to be provided with exact ‘path to product’ in a store(using the communication network and method described with reference toFIG. 50 previously), upon which the VS provides the user with exactcoordinates of the product in the store and directions to get there. Theproduct location and directions may be overlaid on a store/mall map.Also, in virtual stores/malls/tourist locations and other virtual placesdescribed in this document, users may request ‘path to products’, andthey will be provided with product location information and directionsoverlaid on a map of the virtual world. Alternatively, they may bedirected to their destination by a virtual assistant or they maydirectly arrive at their virtual destination/product location in thevirtual world.

Order placements and business transactions can also be conducted via auser's mobile device. A user may view a list of products and services ontheir mobile device. The user may place an order for a product orservice via their mobile device via SMS or other services using the WAPprotocol or through a cell phone based browser in exemplary embodiment.The vendor is informed of the order placed through a web portal andkeeps item ready for pick up or delivers the item to address specifiedby user or the current location of user, which may be determined using acell phone location technique such as GPS and cell-phone triangulation.Users may pre-pay for services or make reservations for services such asthose provided in a salon via their mobile device and save waiting timeat the salon. Vendors may have access to ‘MyStore’ pages, as describedin exemplary embodiment previously with reference to FIG. 42. Once theorder and transaction is approved, a confirmation is sent to the user.Electronic receipts may be sent to the user on their cell phone viaemail, SMS, web mail, or any other messaging protocol compatible withcell phones. Other information can be linked to the cell phone based onelectronic receipts such as warranty and other information as describedpreviously with reference to electronic receipts.

In an exemplary embodiment, a user ‘Ann’ may be a tourist visiting Italyfor the first time, and would like to find out which restaurants havegood ratings and where they are located. The user can query the systemto determine which restaurants ‘Jim’ (a friend who visited Italyrecently) ate at, their locations, and the menu items he recommends. Thesystem, in this case, looks up Ann's friend's network on a socialnetworking site, in exemplary embodiment, to access and query Jim'saccount and acquire the appropriate information. Jim has a virtual mapapplication where he has marked the location of the restaurants hevisited when he was in Italy. The restaurants each have a digitized menuavailable (hyperlinked to the restaurant location on the map) whereitems can be rated by a given user. Given that Ann has permission toaccess Jim's information, the information pertaining to location of therestaurants that Jim visited and liked and the ratings of menu items ofthose restaurants will be made available to Ann on her mobile device.Alternatively, Jim's travel information may be available from a travelitinerary that is in document or other format. In this case, therestaurant location information may be overlaid onto a virtual map andpresented to Ann. The menu items that Jim recommended, along with theirratings may be hyperlinked to the restaurant information on the map indocument, graphics, video or other format. Other files such as photostaken by Jim at the restaurants, may be hyperlinked to the respectiverestaurant location on the map. Thus, in this example, the VS utilizedinformation on a friend's account that may be located on a user'smachine or other machine on the local network, or on the communityserver 26 or on a remote machine on the internet; a map application thatmay be present on the local machine, or on the portal server 20 or otherremote machine; and restaurant information on the retail server 24 orother machine. In this manner, the VS can combine information and dataand/or services from one or more storage devices and/or from one or moreservers in the communication network in FIG. 51A.

Users may utilize the VS for sharing content ‘on the fly’. A website orspace on a web server may exist where users can create their ‘sharingnetworks’. Alternatively, sharing networks may be created via a localapplication software that can be installed on a computing machine. Asharing network comprises member users whom the user would like to sharecontent with. A user may create more than one sharing network based onthe type of content he/she would like to share with members of eachnetwork. Members may approve/decline request to be added to a sharingnetwork. A space is provided to each sharing network where the membersin the sharing network may upload content via their mobile communicationdevice or a computing machine by logging into their sharing network.Once the user uploads content into the sharing space, all members ofthat particular sharing space are notified of the update. Sharingnetwork members will be informed immediately via an SMS/text messagenotification broadcast, as an example. Members may change thenotification timing. They may also alternatively or additionally opt toreceive notification messages via email and/or phone call. In exemplaryembodiment, a user may upload videos to a sharing space. Once the videohas been uploaded, all the other members of the sharing network arenotified of the update. Members of the network may then choose to sendcomments ‘on the fly’ i.e., members respond to the video update byposting their comments, for which notifications are in turn broadcast toall members of the sharing network. In another exemplary embodiment, theVS may directly broadcast the uploaded content or asummary/preview/teaser of the uploaded content to all members of thesharing network. Real-time communication is also facilitated betweenmembers of a sharing network. Chat messages and live video content suchas that from a webcam can be broadcast to members of a sharing networkin real-time. The sharing network feature may be available as astandalone feature and not necessarily as part of the VS.

The tourism industry can make use of the VS to provide users with guidedtours as the user is touring the site. Instructions such as ‘on yourright is the old Heritage building’, and ‘in front of you are the GreenGardens’, may be provided as the user browses a site and transmitsvisual and/or text and/or speech information via their mobile and/orother computing device to the VS. In exemplary embodiment, a user maytransmit site information in the form of images/videos to the VS, as hebrowses the site on foot. Alternatively or additionally, the VS canprovide tour guide information based on the GPS coordinates of a user.Instructions may be provided live as the user is touring a site. Theuser may transmit their views via a webcam to the tour application,which is part of the VS. The tour application then processes theimages/videos in real-time and transmits information on the what isbeing viewed by the user (i.e., ‘guided tour’ information). Users mayask the VS/tour application queries such as ‘What is this’ and point toa landmark in the image or ask ‘What is this white structure with blacktrimmings to my left?’. Thus, the VS tour application may decipherspeech information and combine the query with image/video and any visualinformation provided to answer the user. The tourinstructions/information can be integrated with whiteboard features sothat landmarks can be highlighted with markings, labels etc., as theuser is touring the site. The VS may alternately or additionallytransmit site information/tour instructions based on the GPS coordinatesand orientation of the user. Orientation information helps to ascertainthe direction in which the user is facing so that appropriate landmarkreferencing may be provided such as ‘to you left is . . . ’, ‘turn rightto enter this 14th century monument’ etc. Orientation may be determinedby observing two consecutive coordinates and computing the displacementvector. Tour information/instructions may be registered with existingmap applications and information and/or street view applications andinformation (for example Google Street View). Computationally intensivetasks, such as registration of the user's view with maps or other viewsin a database, may be transmitted to a remote server and the results maybe transmitted back to the user's mobile device. Advertisementinformation may be overlaid/linked to relevant sites on user views on amobile in exemplary embodiment.

Data from the user's mobile device may be used to reconstruct a 3D modelof the scene, and may be available for viewing remotely. Thereconstruction, if too intensive’ may occur on a remote machine.

Instructions may also be catered to users on foot (instead of in avehicle for example), via the handheld. These include instructionsspecific to a person on foot, such as ‘turn around’, ‘look up’, inexemplary embodiment. In the case of directions to a location as well,users may be provided alternate instructions to arrive at a destinationwhen traveling by foot (thus, directions are not limited to drivingdirections).

The VS may be integrated with a map application where users can directlyor mark recommended places to visit. These marked places may behyperlinked with to-do lists that specify the activities or events theuser can engage in at those places; or blogs that catalogue userexperiences. Photos, videos and other graphics and multimedia contentmay be linked to a place on the map describing the place, itssignificance and its attractions. These may also be pictures/videostaken by friends, virtual tours etc. A user may add or request to seespecific feeds for a given place. In exemplary embodiment, the localnews headlines corresponding to a selected place on the map may bedisplayed. Areas of interest such as general news, weather, science orentertainment, may be selected by the user to filter and display newsand other information of interest. Event feeds that display events oractivities on a particular month or week or day of the year at a placemay be requested. Generic user videos capturing user experience ortravel content at a place may be displayed. These may be videos that areextracted from a video uploading site such as YouTube, based on keywordssuch as place or other default keywords or keywords specified by theuser. Local shopping feeds containing information about the places withthe most popular or cheap and other categories of shopping items may belinked or associated with the places on the map. Most popular localmusic and where to buy information may be associated with a place. Otherlocal information such as car rentals, local transit, restaurants,fitness clubs and other information can be requested by the user. Thus,local information is made easily available on any computing or mobile ordisplay device. In addition, map overlays and hyperlinks to appropriatesources/places are used in order to make information presentation asuser-friendly as possible. The user can also request the VS to displayitineraries that include cities, places, events, attractions, hotelsthat the user chooses. In addition, the user may specify filters such asprice range and time period to include in forming the itinerary. The VSwould scan the appropriate databases detailing places, events,attractions and hotels and their associated information such as prices,availability, ticket information etc. in order to draw up a suggesteditinerary accommodating user requirements as best as possible. The usermay make all reservations and purchases of tickets online. The VS woulddirect the user to the appropriate reservation, purchasing and ticketingagents. Alternatively, the VS may be equipped with a facility to makehotel, event bookings and ticket purchases (for events, attractionsetc.) online.

The VS may be used to connect to the services in a local community aswell. Users can request an appointment at the dentist's office, uponwhich the system will connect to a scheduling software at the dentist'send (service's end), in exemplary embodiment. The scheduling softwarewould check for available slots on the day and time requested by theuser, schedule an appointment if the slot is available and send aconfirmation to the VS. The VS then informs the user of theconfirmation. If the available date and time is already taken or notavailable, the scheduler sends the user a list of available slots aroundthe day and time the user has requested. The VS provides thisinformation to the user in a user-friendly format and responds to thescheduler with the option the user has selected.

A facility is a ‘Centralized Communication Portal’ (CCP) which providesusers with access to all emails (work, home, web based, localapplication based), voice messages, text messages, VoIP messages, chatmessages, phone calls, faxes and any other messages/calls availablethrough electronic messaging services. The CCP may take the form of aweb based software or a mobile device software and/or both and/or localapplication for use on a computing machine or a mobile device or alandline phone. The CCP is equipped with text-to-speech andspeech-to-text conversion so that it is possible for users to accessemails in the form of voice messages, and voice messages in text format,in exemplary embodiment. The user can set the display name and number oremail address of outgoing phone calls, emails, SMS or the system candetermine these automatically based on factors such as who the messageis for or what the context of the message is, etc. The system only letsthe users set the phone number or email address of outgoing messages ifthe user owns these phone numbers and email addresses. In an exemplaryembodiment, the owner ship of a phone number or email address isestablished by posing a challenge question to the user the answer towhich is sent to the phone number or email address.

While a person is on a call, the CCP can simultaneously make a recordingof the conversation, if access is granted by the participants of thecall; convert the call recording into text; reformat the message ifnecessary and provide the user with options to do something with therecording such as email or save call recording, in an exemplaryembodiment. The CCP can keep track of a call or message duration and/orsize. This may be useful in case of professional services that chargeper call or message for their services provided via phone or email orother messaging service(s). The CCP allows users to program features. Inan exemplary embodiment, users can program the CCP to respond in acertain way to an incoming call. For example, the user may program theCCP to ignore call or forward the call to an answering machine, if theincoming call is from a specific number or person, for instance. Inanother exemplary embodiment, a user (Ann, for example) may program theCCP to respond to calls by automatically receiving the call after tworings, for example, and playing a message such as ‘please state yourname’, or ‘please wait until Ann picks up’, or playing audio tracks froma certain folder available on the user's local machine or a remotemachine or through a web page. If the caller user is logged into theirCCP account, available through a web page or a local application ontheir computer or mobile device, then they may be able to view videosthat the receiver user (i.e., the user receiving the call) hasprogrammed the CCP to play before they pick up the call (the video mayplay via a visual interface provided by the CCP). In another exemplaryembodiment of programming options, users may be able to set forwardingoptions for incoming calls and emails. For example, the user may programthe CCP to forward all incoming emails (chat or text messages) orincoming emails (chat or text messages) from specific users to a mobilehandheld/phone; forward incoming calls to a mobile phone to an emailaddress or to another cell phone(s), in exemplary embodiments. Images inemails/text/chat messages may be converted to text using computer visiontechniques such as those described with reference to FIG. 52 and FIG. 6.Text to speech conversion may then be carried out and, thus imageinformation in text/email/chat messages can also be made available viavoice messages or voice chat. PBX (Private Branch eXchange) systems maybe integrated with the CCP.

An easy-to-use visual interface may be provided by the CCP. When a callis made, the interface may display the status of the receiver user. Inexemplary embodiment, the status of a user may be: busy, back in 10minutes, not in, hold/wait, leave message, attending another call, callanother number: #####, etc. In another exemplary embodiment, a virtualcharacter may greet the caller via the visual interface and inform thecaller of the receiver's status, and instruct the caller to leave amessage or direct the caller to another phone number or providealternate directions. In another exemplary embodiment, a video recordingof the receiver user may greet the caller user and provide statusinformation and/or instructions to leave a message, call another number,hold/wait etc. Image to text conversions may also be useful to conveyvisual elements of a conversation (in addition to the audio/speechelements), in the case that users would like to view webcam/videoconversations in text message form or in audio/voice format. Text toimage conversion can be carried out using techniques similar to thosedescribed in [60]. This conversion may be utilized when users opts tosee email/chat/text/SMS messages via the visual interface. In this case,in addition to displaying text information, image information obtainedvia text-to-image conversion may be displayed. Alternatively, thisconverted image information can be displayed as a summary or as asupplement to the actual messages.

Users may additionally connect to each other during a call or chat oremail communication via webcam (s) whose output is available via theCCP's visual interface. Any or all of the collaborative tools, andmethods of interaction discussed with reference to FIG. 20 may be madeavailable to users by the CCP for collaborative interaction betweenparticipants during a call or chat or email communication via the CCP'svisual interface. Users may be able to organize their messages, callinformation and history in an environment that allows flexibility. Inexemplary embodiment, users may be able to create folders and move, add,delete information to and from folders. They may tag messages and callsreceived/sent. They may organize calls and messages according to tagsprovided by the system (such as sender, date) or custom tags that theycan create. Call and message content and tags are searchable. Spamdetection for phone calls, chat, text and voice messages (includingVoIP) is integrated with the CCP, in addition to spam detection foremail. In an exemplary embodiment, this is accomplished using aclassifier such as a Naïve Bayes classifier [7, 61]. In addition, spamfeature lists may be created using input from several users as well asdummy accounts. In an exemplary embodiment, if a user's friend whoreceives the same or similar email, phone call, SMS, etc. marks it asspam then the probability of that message being spam is increased. Dummyaccounts may be setup and posted on various sources such as on theinternet and messages collected on these accounts are also marked with ahigh probability of being spam. Users also have the option to unmarkthese sources/numbers as spam. A signature may be used by the CCP toconfirm the authenticity of the source of the message. In an exemplaryembodiment, this signature is produced when the user's friend logs intothe system. In another exemplary embodiment, this signature may beproduced based on the knowledge of the user's friend available to theCCP. Additionally, the CCP may inform the user that a particular numberappears to be spam and if the user would like to pick up the phoneand/or mark the caller as spam. The CCP may additionally provide theuser with options regarding spam calls such as: mute the volume for aspam call (so that rings are not heard), direct to answering machine,respond to spam call with an automated message, or end call, blockcaller etc. Users may arrange meetings via the CCP. A user may specifymeeting information such as the date, time and location options, membersof the meeting, topic, agenda. The CCP then arranges the meeting onbehalf of the user by contacting the members of the meeting andconfirming their attendance and/or acquiring alternate date, time,location and other options pertaining to the meeting that may be moreconvenient for a particular member. If any of the users is not able toattend, the CCP tries to arrange an alternate meeting using thedate/time/location information as specified by the user that is not ableto attend and/or seeks an alternate meeting date/time/location from theuser wishing to arrange the meeting. The CCP repeats the process untilall users confirm that they can attend or until it synchronizesalternate date, time and location parameters specified by all members ofthe meeting. Users may specify the best mode such as email, phone, fax,voice, chat, text message via which the CCP may contact them to arrangea meeting. Users can also confirm whether they would be attending ameeting in person or via video/phone conferencing etc. Instead ofproviding only a binary classification (“spam” or “not spam”), the spamdetector may provide more levels of spam detection; it may provideseveral levels of classification. If desired by the user, it canautomatically sort emails, phone calls, SMS, etc. based on variouscriteria such as importance, nature (eg. social, work related, info,confirmation, etc.) etc. This may be done in an exemplary embodiment bylearning from labels specified by users and/or attributes extracted fromthe content of the email, phone call, SMS etc. using Naïve Bayes. In anexemplary embodiment, a technique similar to that used in [62] is usedfor ranking.

The CCP may assign users a unique ID similar to a unique phone number oremail address, which may consist of alphanumeric characters and symbols.In exemplary embodiment, it may assume the form ‘username#company’. Itmay be tied to existing top-level domains (TLDs), for example, the‘.com’ domain. When someone dials or types this ID, a look up table isused to resolve the intended address which could be a phone number oremail/chat address or VoIP ID/address or SMS ID. Users may specifywhether they would like to use the CCP ID as the primary address tocommunicate with any user on their contact list. Users may also use theCCP ID as an alias.

The CCP may be integrated with the VS and/or incorporates one or morefeatures of the VS, and vice versa.

An example of a “Job Application and Resume Management Service” (JARMS)is described next. This application may be available on the portalserver 20. Users can create their “Job Profile” via this service. Formsand fields will be available for users to document their background andqualifications including their personal history, education, work andvoluntary experience, extra-curriculars, affiliations, publications,awards and accomplishments, and other information of relevance to theircareers. This service would provide questionnaires that may be useful torecord or test skill subsets of the user. Hiring managers may find thisadditional information useful to assess a given job applicant's skills.Furthermore, commonly asked questions by Human Resources (HR) personnelmay be made available for users to post answers to. This would assistthe employers in further reducing application processing times. Theskill and HR questions may be posted in text, audio, video and any othermultimedia format. The user responses to those questions may also beposted in text, audio, video and any other multimedia format. A“Portfolio” section is available that assists the user in developing,preparing and uploading documents and other files of relevance to theircareer, for example, resumes, posters, publications, bibliographies,references, transcripts, reports, manuals, websites etc. This servicewill make it convenient for the user to upload documents in a variety offormats. Also, the user can design different resumes for application todifferent types of jobs. A tools suite assists the user in documentuploading, manipulation and conversion. In exemplary embodiment, a PDF(Portable Document Format) conversion tool, document mark-up, and othertools are provided to the user. Users can upload transcripts directlyfrom their University Registrar/Transcript offices, or websites throughthis service. The transcripts may be authenticated by the Universitiesor certified electronically. In this manner, the employers can beassured of the validity of the transcript uploaded through this service.References and their contact information is provided by the user viathis service. Links to the accounts of the referees on JARMS or socialnetworking sites such as Linkedin may also be provided on the user'sprofile. Videos from YouTube or other sources that document useraccomplishments or work such as a conference presentation or an onlineseminar or a product demonstration and other examples may be uploaded.

JARMS is equipped with additional security features so that informationis not easily viewed or captured by third party individuals or softwareetc. Employers to which users are interested in submitting theirapplication to may be provided with access to the user's job profile.Users may also select the account resources they would like to makeaccessible to the employer.

An “Interview Room” facility is available through JARMS which is anonline space where real time interviews can be conducted. Visualinformation along with audio and other content from a webcam, camcorder,phone etc. may be broadcast and displayed in windows that assume aconfiguration as shown in FIG. 53, so that all users in an interviewsession can be seen simultaneously. The interview room may be moderatedby personnel from the institution or company that is conducting theinterview. This session moderator can allow or disallow individuals fromjoining the session. The interviewee and interviewers can view eachother simultaneously during the interview session in the display windowsin FIG. 53, by using video capture devices at each end and broadcastingthe captured content. The interview may involve video and audio contentonly or it may be aided by speech to text devices that convert audiocontent to text and display content as in the ‘Transcript’ display boxFIG. 53. Alternately, text input devices such as a keyboard/mouse may beused to enter text. JARMS sessions may be private or public. Thesesessions may be saved or loaded or continued or restored. The sessioncontent including video content may be played, paused, rewinded,forwarded.

The collaborative broadcasting and viewing of content in windowsarranged as in the configuration given in FIG. 53 may occur during anonline shopping session or during a news coverage session online or atechnical support session and during other collaborative communicationand broadcast sessions online. In exemplary embodiment, during a newsbroadcast session, questions posed by viewers of the news story willappear in a ‘Live Viewer Feed’ (Feedback) box. Another feature, “LiveImage Retrieval” looks up/searches for images corresponding to the wordsrelayed in the broadcast in real-time, either on the local machine orthe internet or a file or folder specified by one or more of the usersinvolved in the collaborative session, and displays the appropriateimages during the session to the viewers in another display window. Thesystem may look up image tags or filename or other fields characterizingor associate with the image in order to perform the image search andretrieval corresponding to words in the collaborative conversation orbroadcast. In exemplary embodiment, this can be accomplished as shown in[60]. The Live Image Retrieval (LIR) application can be used with otherapplications and in other scenarios. In exemplary embodiment, a user mayspecify an object in text or voice or other audio format, during onlineshopping. The LIR would retrieve images corresponding to the specifiedword from the retail server 24. The user can then select the retrievedimage that best matches the user's notion of that object. For instance,the user may specify black purse and the LIR would retrieve images ofmany different types of black purses from different sources such as ablack leather purse, brand name/regular black purses, black purses instores in proximity of the user's location, fancy/everyday use blackpurses, etc. When the user's selects the purse meeting characteristicsthat the user is looking for, system 10 or the VS directs the user tothe source of that purse, which may be an online store.

Another application (‘Social Bug’—SB) in the portal server 20 isdescribed next that lets users upload content conveying information ofinterest to the general public such as activities, restaurants,shopping, news etc. These topics may be linked to specific geographicalareas, so that users can look up information that pertains to a specificregion of interest, such as the local community they reside in. So, inexemplary embodiment, users may look up or search content related toactivities and events in their local community. The content may beuploaded by common users or business owners. Such video content willprovide more information related to a topic in the form of reviews, userexperiences, recommendations etc. The content is as dynamic and topicsas wide-ranging as the users' interests. The uploaded content may assumethe format of videos in exemplary embodiment. Moderators for each regionmay filter the content uploaded by users and choose the most relevantvideos. The content may be organized or categorized according to fieldssuch as ‘activities’, ‘events’, ‘businesses’, ‘shopping item/store’,‘news area’ etc. Users can also specify the kind of information theywould like to receive more information on via feeds, in an exemplaryembodiment. Users may opt to receive feeds on a particular tag/keywordor user or event or business or subject.

The user can indicate specific filters like ‘video author’, ‘reviewer’,‘subject’, ‘region/locality’, ‘date created’, ‘event date’, ‘pricerange’, and videos, video feeds and related content will be presentedgrouped according to the filters and/or filter combinations and keywordsspecified. Users can also specify objects in videos they are lookingfor, for example, ‘Italian pasta’, or a particular chef, in videos aboutrestaurants. Video tags and other information describing a video (suchas title, author, description, location etc.) may be used in order tofind and filter videos based on criteria specified by the user.Additionally, video content (for instance, image frames, music andspeech content) is mined in order to filter or find videos according tothe user specified criteria.

This application allows users to indicate whether they liked a givenvideo. Users can specify what they like about a video using keywords.Users may specify what kind of content they would like to see more of. Asection/field titled ‘More of . . . ” would assist users in specifyingpreferences, suggestions about content they like or would like to seemore of.

Relevant links and applications would be provided to users via thisservice depending on the content being viewed. In exemplary embodiment,if users are viewing restaurant related content, links would be providedallowing users to send a query to the restaurant, call up therestaurant, or book reservations via SMS, phone, email or chat.Similarly, if the user is viewing news items, news feed items and pollsrelated to the content the user is viewing will be provided in the formof summaries or links. Top rated or most viewed response videos postedby viewers to news stories may also be posted on the same page. Videosmay be pre-filtered by moderators. In exemplary embodiment,organizations working for social causes can post response videos to newsstories covering issues such as poverty or human rights. They mayconduct campaigns or provide information online through the use ofvideos. Such response videos will help to target specific audiencesinterested in the issues the organization is working/campaigning for.Since news videos are more popular, traffic can be directed to othervideos relaying similar content but which may not necessarily belong tothe same genre (for instance, two videos may both talk about poverty,but one may be a news story and the other an advertisement ordocumentary produced by an NGO). These videos may be posted as responsevideos to more popular videos, which may not necessarily be news videos.

Objects in videos and/or frames may be hyperlinked and/or tagged. Inexemplary while browsing a jewelry store advertisement or infomercial, auser may click or hover or select an item of interest (a necklace, forexample) and be provided with details on the make, model, materials ofthe necklace, pricing information etc. on the same or differentframe/page. Alternatively/additionally, while a user is browsing thevideo, tags/comments/links may appear automatically. Users may also beprovided with additional information such as deals available at thestore; other users browsing the video and user's friends, if any, thatare browsing/have browsed the same video or shopped at the store; wheresimilar products or items may be found; store/businessratings/comments/reviews; how the store compares with other stores withreference to specific criteria such as bargains, quality, service,availability of items, location accessibility. Additional features suchas those discussed with reference to FIG. 36 may be available. Inanother exemplary embodiment, tagged/hyperlinked objects withinvideos/images/simulations (which may be live or not) may be used forproviding guided tours. In another exemplary embodiment, videos/imageframes may be tagged/hyperlinked. As a video plays and a tagged frameappears, the corresponding tag is displayed to the user. Thetags/hyperlinks/comments described above are searchable. On searchingfor a tag or browsing through tags the corresponding videos are shown.

Users can also avail of the translation feature that enables translationof videos in different languages either in real-time or offline. Text,audio and/or video content is translated and presented as audio/speech,text (subtitles for example). Shared viewing of videos between friendsis possible. When shared viewing or broadcasting occurs, the same videomay be simultaneously viewed by users sharing it, in differentlanguages. The same feature is available in any/all of the chatapplications mentioned in this document i.e., text typed in a certainlanguage in a chat application may be translated to multiple languagesand made available in real-time or offline to the different users of achat session in audio/speech, text (subtitles for example). The videopresentation/content may be interactive i.e., users watching the videosmay interact with each other via the collaborative tools described withreference to FIG. 20 and modes referenced in FIG. 7. Additionally thevideo may be a live broadcast where the presenter or video author(s) orvideo participants may interact with the audience watching the broadcastvia the collaborative tools described with reference to FIG. 20 andmodes referenced in FIG. 7.

Summaries of video content may be provided in addition to video lists.Conferences or seminars or news stories or documentaries or movies maybe summarized and provided to users. Users may be able to obtain areal-time summary of a given video before choosing to view the completevideo. Composite summaries of related videos or videos grouped bysubject, tags or title or author or keyword or any other criteria may beprovided to users. This involves providing a summary of all videos inthe group in one video. As the composite video plays, individual linksto the corresponding video being shown in the summary at any givenmoment, are displayed. Video summarization (VSumm) techniques mayinvolve tracking of most popular keywords. These include most commonlyused search terms, and tags of most viewed videos in exemplaryembodiment. VSumm may also keep track of important keywords via phrasesimplicitly referencing them such as ‘important point to be noted is . .. ’ in a video, in order to identify important regions/content in videos(i.e., these regions are namely those audio/video signal sequences in avideo in which important keywords are embedded).

Additionally, users may specify summarization parameters, such as thelength of the summarized video and/or filters. Users can employ filtersto specify scenes (video, audio, text content/clips) to include in thesummaries. These filters may include keywords or person or object namecontained in the video clip to be included in the summary. In exemplaryembodiment, a user may specify an actor's name whose scenes are to becontained in the summary of a movie. Other filters may include the kindof content the user would like to pre-filter in the video such as‘obscene language’ in exemplary embodiment.

Given a video/audio/text sequence, the sequence can be summarizedaccording to the procedure illustrated in FIG. 55 and described next, inexemplary embodiment. Given an audio-visual A/V (or audio, or image orvideo, or text or any combination thereof) sequence, it may be brokendown (split) into audio, video, image and text streams, whilemaintaining association. In exemplary embodiment, if a PowerPointpresentation is the input, then the audio-video-image-text content onany given slide is associated. If an audio-video sequence is beinganalyzed, then audio and video signals at any given time are associated.Different processing techniques are then applied in different stages asshown in FIG. 55 to carry out the input sequence summarization.

At the optional Filtering step, pre-processing is carried out usingdigital signal processing techniques. In an exemplary embodiment, atransformation is applied to an image sequence to convert it into thecorresponding signal in some pre-defined feature space. For example, aCanny Edge detector may be applied to the frames of an image sequence toobtain an edge space version of the image. Multiple filters may beapplied at this step. Subsequences can be identified not just over time,but also over frequency and space. The resulting pre-processed datasequences are passed on to the Grouping stage.

At the Grouping stage, subsequences are identified and grouped based ontheir similarity. Distance metrics such as Kullback-Leibler divergence,relative entropy, mutual information, Hellinger distance, L1 or L2distance are used to provide a measure of similarity between consecutiveimages, in exemplary embodiment. For instance, when mutual informationis computed for consecutive data frames, and a high value is obtained,the data frames are placed in the same group; if a low value isobtained, the frame is placed in a new group. Motion information is alsoextracted from an image sequence using optical flow for example.Subsequences exhibiting similar motion are grouped together. Frequenciescorresponding to different sources, for example different speakers areidentified and may be used during synopsis formation. For instance, ascript may be composed based on users identified and their spoken words.In exemplary embodiment, frequencies corresponding to different sourcesare identified using expectation-maximization (EM) with Mixture ofGaussians (MoG). This method may also be used in the context ofinterviews (as described with reference to FIG. 53), live broadcasts,and other video and data sequence summaries.

Semantic analysis is then carried out on the data sequence to identifyand localize important pieces of information within a subsequence. Fortext information, for instance, large-font orbold/italicized/highlighted/underlined and other specially formattedtext, which generally indicates highlighted/important points, isidentified. Significant objects and scenes within an image or videosequence, may be identified using object recognition and computer visiontechniques. Significant speech or audio components may be identified byanalyzing tone, mood, expression and other characteristics in thesignal. Using expectation-maximization (EM) with Mixture of Gaussians(MoG) for example, the speech signal can be separated from backgroundmusic or the speech of a celebrity can be separated from backgroundnoise.

If the input information is associated with a tagged file, such as anXML file for example or the file shown with reference to FIG. 37, thentags may be analyzed to identify important components. In exemplaryembodiment, in the case of a text file, the associated tagged filedescribing the text may contain tags indicating bold/italicized pointsi.e., important content in the file. From subsequences determined to besignificant, exemplars may be extracted. Exemplars may be a portion ofthe subsequence. For example, in the case of text, it could be a word ora sentence; for an image sequence it could be a frame or a portion ofthe frame or a set of frames or a composite of frames/frame portions inthe subsequence; for an audio signal it could be a syllable(s), or aword, or a music note(s) or a sentence (this system also enables musicto text conversion. Notes corresponding to the music may be output as atext file. For example, it may contain C-sharp, A-minor). Thesubsequences may additionally be compressed (lossless or lossycompression may occur) using Wavelet transform (for example),composited, shortened, decimated, excised or discarded. Thissummarization procedure is also useful for mobile applications wherebandwidth, graphics and memory resources are limiting.

In another exemplary embodiment, an image can be divided in space intodifferent regions and the most significant components can be extractedbased on an evaluation of the significance of the information in theseregions. In yet another exemplary embodiment, significant components canbe extracted from a sequence of images, and these significant portionscan then be composited together within a single image or a sequence ofimages, similar to a collage or mosaic.

In exemplary embodiment, in FIG. 55 the sequence represents an inputdata sequence (each square represents a single frame or data unit in theinput information sequence). The sequence may consist of differentscenes. For example, a given scene could be one that represents theinside of a car; another could be an office scene shot from a particularviewpoint; another could be a lecture slide. At the Grouping step,subsequences are identified based on similarity measures describedbefore. The different subsequences that are identified by the algorithmare shown with different symbols in this figure. Subsequences can be ofvariable length as illustrated in FIG. 55. The Semantic analysis stepthen extracts exemplars from each group (in this case +, O). In thiscase, the algorithm picks out a + frame from the subsequence it labeledas ‘+’, and a portion (O, O) of the subsequence it identified as ‘O’.

The associated data—audio, video sequence data are reformatted. Inexemplary embodiment, reformatting is based on significance. Forinstance, if an image is larger, it may occupy a larger portion of thescreen. Audio content may be renormalized if necessary. The audio, videoand text channels may be merged to produce a new sequence or they may beprovided to the user separately without merging.

The AFMS, VS, LIR, JARMS, SB systems may be used within a local areanetwork such as a home or office network. Users who wish to share eachother's data may be added to the network permitting sharing ofapplications within the network and restricting access to the data ofthe shared network users. The AFMS, VS, LIR, JARMS, SB systems and/orthe features and methods described with reference to system 10 and/or acombination of any of the above may be used in conjunction with eachother or independently. One or more features and methods of the AFMS,VS, LIR, JARMS, SB systems and/or the features and methods describedwith reference to system 10 and/or any combination of the above may beused as standalone features as part independent systems or as part ofother systems not described in this document.

The shopping trip feature may be incorporated as a feature that is partof a browser or that may be installed as a browser plug in. This wouldallow activation of the shopping trip upon visiting almost any siteaccessible by the browser. All of the features described as part of thisinvention can also be incorporated as such i.e., as part of a browser oras a browser plug in, making it possible to use these features on anysite.

This invention further illustrates the 3D browser concept. This browserwould incorporate web pages and websites with the depth component inaddition to 2D elements. Users will be able to get a sense of 3D spaceas opposed to 2D space while browsing web pages and websites via the 3Dbrowser.

This invention incorporates additional features available on a mobiledevice such as a mobile phone or a personal digital assistant (PDA) toassist the user while shopping in a physical store. When users enter astore, the mobile device will detect and identify the store by receivingand processing wireless signals that may be sent by a transmitter in thestore, and will greet users with the appropriate welcome message. Forexample, if the store is called ‘ABC’, the user will be greeted with themessage ‘welcome to ABC’ on their wireless device. The user may beuniquely identified by the store based on their mobile phone number forexample. The store may have a unique ID that will identified by the cellphone and used to also keep track of stores/places visited by the user.Additionally, store specials and offers and other information may bepresented to the user on their mobile device (in the form of visual oraudio or other forms of relaying digital input on a mobile device).Instead of automatic store identification, the mobile may instead acceptuser input (text, speech and other forms) for identifying store and thenpresent relevant store information to the user. Users will be able tosearch for items in the store using their mobile device and will be ableto identify the location (such as the department, aisle, counterlocation etc.) of the product they wish to buy. They will receive anindication of whether they are approaching the location of or are in thevicinity of the product in the store and/or if they have reached oridentified the correct location. The user may see a ‘path to product’ asdescribed elsewhere in this document. The mobile device is equipped witha barcode scanner and can be used for checking inventory, price andproduct information by scanning the barcode on a product. The mobiledevice may also process the user's shopping list available on the mobiledevice and automatically generate availability, inventory, location,discounts, product description, reviews and other relevant informationpertaining to the product and display it to the user. In an exemplaryembodiment, this may be accomplished as follows with reference to FIG.50. The mobile device 901 may transmit appropriate informationrequest/query signals to a wireless SAP (service access point) in thestore which in turn, will transmit relevant store and productinformation which is received and displayed by the mobile device.Depending on the specific area of the store that the user is in, theproducts in that area may be displayed on their mobile device. Users mayalso access their model on their mobile device and try-on apparel on themodel, via a local application 271 version for mobile devices. A usermay also go on a shopping trip (as discussed with reference to FIG. 20)using their mobile phone 901. Other members of the shopping trip may beusing a mobile device 902 as well or a computer. Users will also be ableto see whether their friends are in the store using their mobile device901.

Reference is now made to FIG. 52A where an image/video/audio/textanalysis module 1550 is shown in an exemplary embodiment. Theimage/video/audio/text analysis 1550 module outlines the steps ofinteraction or engagement with the outside world, i.e. external to thecomputer. The module 1550 may be used for generic image/audio/video/textscene analysis. In an exemplary embodiment, this module works asfollows: The module is preloaded with a basic language that is stored ina “memory” database 1554. This language contains a dictionary which inturn contains words and their meanings, grammar (syntax, lexis,semantics, pragmatics, etc.), pronunciation, relation between words, andan appearance library 1556. The appearance library 1556 consists of anappearance based representation of all or a subset of the words in thedictionary. Such a correspondence between words or phrases, theirpronunciation including phonemes and audio information, and appearancesis established in an exemplary embodiment using Probabilistic LatentSemantic Analysis (PLSA) [55]. In an exemplary embodiment graphs (a setof vertices and edges) or cladograms are used to represent the relationbetween words. Words are represented by vertices in the graph. Wordsthat are related are connected by edges. Edges encode similarity anddifferences between the attached words. A visual representation of thesimilarity could be made by making the length of the edges linking wordsproportional to the degree of similarity. Vertices converge and divergeas more and more information becomes available. (For example, if thesystem is only aware of shoes as something that is worn on feet, and itlater comes across the word or a picture of sneakers, it may group itwith shoes. As it learns more related words such as slippers or sandals,it groups them together but may later create separate groups for each onlearning the differences between these apparel) This system also enablesconversion from speech to image, image to speech, text to image, imageto text, text to speech, speech to text, image to text to speech, speechto text to image or any combination thereof. The memory database 1554and the appearance library 1556 are analogous to “experience”. Theappearance library 1556 and the memory database 1554 may be used duringthe primitive extraction, fusion, hypothesis formation, sceneinterpretation, innovation, communication, and other steps to assist theprocess by providing prior knowledge. Shown on the right in FIG. 52A arethe steps of analysis of stimuli from the external world. The stimulican be images, video, or audio in an exemplary embodiment. It could alsoinclude temperature, a representation of taste, atmospheric conditions,etc. From these stimuli basic primitives are extracted. More complexprimitives are then extracted these basic primitives. This may be basedon an analysis of intra-primitive and inter-primitive relations. Thismay trigger the extraction of other basic primitives or complex filtersin a “focus shifting” loop where focus of the system shifts from oneregion or aspect of a stimulus to another aspect or region of thestimulus. Associations between the complex primitives are formed andthese primitives are then fused. (The primitive extraction and fusionmethod described here is similar to that described in reference to FIG.6D for the case of images and video. The prior knowledge 112 isavailable as part of the appearance library 1556 and the memory database1554. The method is also applicable for audio stimuli). Hypotheses arethen formed and are verified. The output of this step is a set ofhypotheses (if multiple hypotheses are found) that are ranked by thedegree of certainty or uncertainty. For example, the output of analysison an image of a scene containing people may be a probability density onthe location of people in the scene. The modes or the “humps” in thisdensity may be used to define hypotheses on the location of people inthe image. The probability of each mode (obtained for example bycomputing the maximum value corresponding to the mode or the mean of themode) may be used to define the certainty of the existence of aninstance of a person at the specified location. The variance of eachmode may be used to define the spatial uncertainty with which a personcan be localized. The output of the hypothesis formation andverification step is passed on to a scene interpretation step at whichthe information makes interpretations on the scene. For example, if thesystem identifies a cow, some chickens, and a horse in a video, andidentifies the sound of crows, it may identify the scene as a farmscene. This may be done using a classifier as described before. Theoutput of the scene analysis step is passed on to an innovation step. Atthis step the system innovative remarks to the analyzed stimuli. In anexemplary embodiment, the system looks for things it has seen in therecent past, surprising things, things of interest for example gadgetsand makes comments such as—“Hey, I saw this guy last week”, “That's thenew gadget that came out yesterday”, or “That's a pleasant surprise”.Surprise is detected using the method described with reference to FIG.52B. At the innovation step, the system also filters out things that itdoes not want to communicate with the outside world. This could includeinformation that is obvious or that which is confidential. The output ofthe innovation model is communicated to the external world. This can bedone via text, audio (using text to speech techniques), images [60] orvideo. The text/audio output may include expressions such as, “I amlooking at a farm scene. There are many farm animals here. I am lookingat the cow. It is a very tiny cow. The crows are trying to eat the corn.The dog is barking . . . ”, and so on. If the system has the capacity toperform physical activities, it may communicate by interactingphysically with the environment. For example, it may pick up an objectit likes and view it from other angles and weigh it. The module 1550 maybe driven by an intention. The intention can be based on the user'sinterest. For example, if the user likes hockey, it may pay moreattention to things that are related to hockey in the stimuli. If thestimulus is a news article that mentions that a new hockey stick by thename “winstick” is out in the market, the module may perform a search onthe “winstick” and extract pricing and availability information and sometechnical details on how the “winstick” is made to be a better hockeystick.

Reference is now made to FIG. 52B where a method 1650 for detectingsurprise is shown in an exemplary embodiment. In an exemplaryembodiment, the method 1650 operates as follows: The method constantlypredicts the state of the system and observes the state of the system.(Alternatively, the method may predict and observe the state only asnecessary). The state of the system includes variables that are ofinterest. For example, the state may include the state of the user whichmay involve the location of the user in a given camera view or the moodof the user extracted from an image or based on the music the user islistening to, or the location of the user extracted from a GlobalPositioning System GPS, the mood of the user's friends, etc. Similarly,the state of the environment may include the weather, the day of theweek, the location where the user is, the number of people at the user'shome, etc. One stage of the predict-update cycle is shown in FIG. 52B.At the i^(th) stage, the system uses the output of the (i−1)^(th) stagei.e. previous stage's output and predicts the state of the system at theprediction step 1652. This can be done, in an exemplary embodiment,using a prediction algorithm such as Gaussian process regression forexample as used in [51] or other statistical approaches such as thoseused in [63]. The output of the prediction stage includes a predictedprobability density of the state of the system. This is passed on to anobservation step 1654 together with an observation of the system. Theoutput of the observation step 1654 includes an updated probabilitydensity called an observed density. An observation of the system, in anexemplary embodiment could be an analysis of an image taken through awebcam (eg. image-based extraction of the pose of the user) or ameasurement of the temperature of the room using a thermal sensor, orany other measurement appropriate for the system. In an exemplaryembodiment, an observed probability density is computed from theobservation and the predicted density by computing the a posterioridensity using Bayes rule. In another exemplary embodiment, the observeddensity is computed based on the observation alone. The differencebetween the predicted probability density and the observed probabilitydensity is then measured at the measurement step 1656. This is done, inan exemplary embodiment, using a distance metric such as theKullback-Leibler divergence or relative entropy, mutual information, theHellinger distance, or the L₁ or L₂ distance. Other statistics orfunctions drawn from the predicted and observed (or updated) probabilitydensities (or distributions) could also be used. At step 1658, a test ismade to determine if the distance is significant. In an exemplaryembodiment, this is done based on a threshold—if the distance is over athreshold, the distance is considered significant and if it is below thethreshold the distance is considered insignificant. The threshold couldbe assigned or could be determined automatically. In an exemplaryembodiment, the threshold is chosen to be a statistic of the predictedor observed density, In another exemplary embodiment, the threshold ischosen to be a function of the degree of certainty or uncertainty in theestimate of the predicted or observed densities. In yet anotherexemplary embodiment, the threshold is learnt from training data. If thedistance is significant, the system is enters a “surprised” state.Otherwise it remains in an “unsurprised” state. The “surprised” stateand the “unsurprised” states are handled by their respective handlers.The degree of surprise may be dependent on the distance between thepredicted and observed probability densities. This allows the system toexpress the degree of surprise. For example, the system may state thatit is “a little surprised” or “very surprised” or even “shocked”. (Overtime if an event becomes common or occurs frequently the system mayincorporate the nature of the event at the prediction step thus leadingto a predicted density that is closer to the observed density andessentially getting used to the event). Such a system is used, forexample, for detecting anomalies. As discussed with reference to FIG.51A, the system may monitor the locations of kids of a home by usingsignals from their cell phones (for example, text messages from theircell phones indicating the GPS coordinates) using a particle filter. Ifa surprise is observed (for example if the location of the kid isoutside the predicted range for the given time), the surprise handlermay send a text notification to the kid's parents. The system may alsobe used in surveillance applications to detect anomalies. As anotherexample, the system may monitor a user's location while he/she isdriving a vehicle on the highway. If the user slows down on the highway,the system may lookup weather and traffic conditions and suggestalternative routes to the user's destinations. If the user's vehiclestops when the system didn't expect it to, the system's surprise handlermay say to the user things such as—“Do you need a tow truck?”, “Iseverything ok?”, “Do you want to call home for help?”, etc. If aresponse is not heard, the system's surprise handler may notify theuser's family or friends. Such a system, may also be used to predict thestate of the user, for example, the mood of the user. If the systemnotices that the user is depressed, the surprise handler may play acomedy video or play a joke to the user to cheer him up. If the user ison a video sharing site or in the TV room for extended hours and thesystem sees that an assignment is due in a couple of days, the systemmay suggest to the user to start working on the assignment and maycomplain to others (such as the user's parents) if the user does notcomply. Such a system is also useful for anomaly detection at a plant.Various parameters may be monitored and the state of the system may bepredicted. If the distance between the predicted and observed states ishigh, an anomaly may be reported to the operator. Images and inputs fromvarious sensors monitoring an inpatient may be analyzed by the systemand anomalies may be reported when necessary. Another application ofmethod 1650, would be as a form of interaction with the user. The methodmay be used to monitor the activities of the user which may be used tobuild a model of the users activities. This model can then be used topredict the activities of the user. If a surprise if found, the surprisehandler could inform the user accordingly. For example, if the user'scalendar says that the user has an appointment with his/her doctor andthe user typically goes to the doctor on time, but on one instance isnot on his/her way to the office (the system may have access to theuser's GPS location and time of arrival from the current location to thedoctor's office or may gather this data from indirect sources such as achat session with the user's friends indicating that the user is goingto be at a friend's party) the surprise handler may state that the useris supposed to be at the doctor's office and is getting late. Thesurprise handles may make similar comments on the user's friend'sactivities. The surprise handler may also take actions such as make aphone call, turn off the room's light if the user falls asleep and wakeup the user when it's time to go to school. Method 1650 also enables asystem to make comments based on visually observing the user. Forexample, the a system may make comments such as, “Wow! Your eye color isthe same as the dress your are wearing”, or “You look pretty today”,based on the user's dressing patterns, method 1650, heuristics thatdefine aesthetics and/or the method used to determine beauty describedearlier in this document. The probability densities referred to abovecan be discrete, continuous, or a sampled version of a continuousdensity or could even be arbitrary functions or simply scalars that arerepresentative of the belief of the state in exemplary embodiments.There may be cases where the system may expect a surprise, but asurprise is not found. In such situations the systems may express thatit is not surprised and explain why. For example, if a tennis playerloses, the system may say that it is not surprised because the wind wasblowing against her direction during the match or if a football teamloses, the system may express to the users that it is not surprisedbecause the positions of the team players was consistentlyill-positioned. As another example, the system may parse news and if itis found that a famous person is dead, it may express that is “shocked”to hear the news. This expression by the system can be made through anumber of ways, for example through the use of text to speechconversion. The concept of surprise can also used for outlier rejection.For example, a system may employ the method described here duringtraining to identify outliers and either not use them or assign lowerweights to them so that the outliers do not corrupt the true patternsthat are sought from a data.

The concept of a clique session is introduced here. A session is alasting connection typically between a client (eg. 14) and a server (eg.20) that is typically initiated when a user is authenticated on theserver and ends when a user chooses to exit the session or the sessiontimes out. On the other hand, a clique session is one in which multipleusers are authenticated and share the same session. A clique session maybe initiated by any subset of the set of users who have agreed tocollaborate or it may require authentication of all the users.Similarly, a clique session can be terminated if any subset or all theusers of the clique session exit. The order of authentication may or maynot be important. In an exemplary embodiment, all users of a cliquesession may have the same unique clique session ID under which theclique session data is stored. Clique sessions are useful for onlinecollaboration applications. Clique session IDs, can also be used foraccessing resources that require high security. For example, users of ajoint account online may choose to have access to the online resourceonly if both users are authenticated and log in. As another example, auser of a bank account may have a question for a bank teller about hisaccount. In order for the teller to view the user's account, the tellerwould first have to log in and then the user would have to log in to thesame account to allow the teller to view the user's account and answerhis question. Clique sessions may also be used for peer-to-peerconnections.

Reference is now made to FIG. 54A-F where novel devices for interactionare shown in exemplary embodiments. These devices allow another way forusers to communicate with computing devices 14. Reference is now made toFIG. 54A where a novel pointing devices are shown in exemplaryembodiments. This could also take a 1D form 1700, a 2D form 1710, or a3D form 1720. In an exemplary embodiment, the 1D form 11700 works asfollows: A source or a transmitter bank 1712 is located on one side ofthe device and a sink or sensor or a receiver bank is located on theopposite side 1714. The source may emit lasers or other optical signals,or any other directional electromagnetic radiation or even fluids. Whenthe beam is interrupted the by an interrupting unit such as a finger ora pen, the corresponding sensor on the receiver bank is blocked fromreceiving the signal. This is used to define the location of the object.If lasers are used, a laser frequency different from that of typicalbackground lighting is used. In an alternative embodiment, theinterrupting unit emit instead of the source or transmitting bank. Theunit also allows the use of multiple interrupting units. In this case,the multiple sensors would be blocked and this would be used to definethe location of the interrupting units. In an alternative embodiment,along each side of the device, a transmitter and receiver may be used inan alternating fashion so that each side has both transmitters andreceivers. In the 2D form 1710, a second set of receivers andtransmitters are placed orthogonal to the first one. Similarly, in the3D form 1720, three sets of transmitter and receiver banks are used.Reference is now made to another pointing device 1730 in FIG. 54A thatis composed of a set of holes. In each of these holes, a transmitter anda receiver are located. Each of these transmitters may employ lasers orother optical signals, or any other directional electromagneticradiation or even fluids. The transmitter and the receiver are bothoriented such that they point out of the device in the direction of thehole. When a hole is covered by an interrupting unit such as a pen or afinger, the signal bounces off the interrupting device and is sensed bythe receiver. This signal is then used to define the location of theinterrupting unit. In all cases 1700, 1710, 1720, 1730 a sequence ofblocked sensors over time can be used to define the direction of motion.Reference is now made to FIG. 54B where an illustration 1732 of the useof the 2D form 1710 is show. The user can simply drag a finger on theunit and use that to point to objects or for free form drawing. The unitmay also be placed over a computer screen and used as a mouse. Alsoshown in FIG. 54B is an illustration 1734 of the use of the 3D form1720. This can be used to manipulate objects in 3D. For example, thiscan be used with the technology described with reference to FIG. 36.This device may be used with a hologram for visual feedback or it may beused with any conventional visualizing unit such as a monitor. Thedevice 1720 can also be used with multiple hands as shown in theillustration 1734. Reference is now made to FIG. 54C where anotherillustration of the use of the device 1710 is shown in an exemplaryembodiment. The device 1710 may be placed on paper and the user may usea pen to write as usual on the paper. As the user writes. the device1710 also captures the position of the pen. This is then used to createa digital version of the writing and may be stored on the unit 1710 ortransferred to a computing device. The device 1710 is also portable. Thecorners of the device 1710 can be pushed inwards and the unit folded asshown in FIG. 54C. The compact form of this device takes the form of apen as shown in FIG. 54C. The device 1710 can also include a palettethat includes drawing tools such as a polygons, selection tools, eraser,etc. The user can also slide the device 1710 as he/she writes to createa larger document than the size of the device. This movement of thedevice 1710 is captured and a map is built accordingly. The motion maybe captured using motion sensors or using optical flow [64] if the unitis equipped with optical sensors. The device 1710 may also be movedarbitrarily in 3D and the motion may be captured along with location ofthe interrupting device to create art or writing in 3D using the 2D form1710. The device 1710 can also be used as a regular mouse. The apparatuspresented in FIG. 54A-C may also be used as a virtual keyboard. Regionsin the grid may be mapped to keyboard keys. In one exemplary embodiment,a user can place the apparatus on a printout of a keyboard (or a virtualkeyboard may be projected using for example lasers) and use it fortyping.

Reference is now made to FIG. 54D where a novel device 1740 forinteracting with a computing device or a television in shown in anexemplary embodiment. The device 1740 includes a QWERTY keyboard or anyother keyboard 1748 that allows users to enter text or alphanumerics, amouse 1746, controls for changing the volume or channels 1744, othercontrols for switching between and controlling computing devices andentertainment devices such as a DVD player, a TV tuner, a cable TV box,a video player, a gaming device. The device may be used as a regularuniversal TV remote and/or to control a computer. The mouse may be usedby rocking the pad 1746 to a preferred direction or sliding a fingerover the pad. The device 1740 communicates with other devices viainfrared, Bluetooth, WiFi, USB and/or other means. The device 1740allows users to control the content being viewed and to manipulatecontent. For example, the device 1740 allows users to watch videos on avideo sharing site. Users can use the keyboard 1748 to enter text in abrowser to go to a site of their choice and enter text into a search boxto bring up the relevant videos to watch. They can then use the mouse1746 to click on the video to watch. The keyboard 1748 and the mouse1746 can be used as a regular keyboard and mouse for use with any otherapplication as well. The keyboard may also be used to switch TV/cablechannels by typing the name of the channel. A numeric keypad may bepresent above the keypad, or number keys may be a part of the alpha(alphabets) keyboard and can be accessed by pressing a function key, inan exemplary embodiment. The device 1740 ma also include an LCD screenor a touch screen. The device 1740 may also be used with a stylus. Thefunctionality of the device may be reprogrammable. The device could alsobe integrated with a phone. The device may be used with one hand or twohands as shown in FIG. 54E in an exemplary embodiment. The device allowseasy text entry when watching videos. The device facilitates interactivetelevision. The content of the television may be changed using thisremote. The device 1740 may also include motion sensors. The motion ofthis device may be used change channels, volume, or control characterson a screen. The device may be used to search a video for tags and jumpto tags of interest. The device may also feature a numeric keypad thatallows easy placement of phone calls.

Reference is now made to FIG. 54F where of a novel human computerinterface system is illustrated in an exemplary embodiment. This systemmakes use of a line of sight that includes two or more objects. In anexemplary embodiment, the location of the user's finger and an eye areused to determine the location where the user is pointing. The locationof the user's finger(s) or hand(s) and that of one or both of the user'seyes can be used to determine where the user is pointing on the screen.The user may point to a screen 1760 using one or more finger(s)/hand(s)1762. One or more cameras may monitor the location of 1762 and theuser's right eye 1764 and/or left eye 1766. The cameras may be on top ofthe screen, on the sides, at the bottom or may even be behind the screen1760. A side view and a top view of the setup are also shown in FIG.54F. The system may make use of motion parallax to precisely determinethe location pointed at by the user.

A feature to enhance user experience with documents (for example on theinternet) is described below. This feature is referred to as a“quotation system”. This feature allows users to quote from documents.In an exemplary embodiment, documents may be uniquely identifiable. Thismay be done by assigning a unique identification number to each documentthat is registered in a database. Documents can be indexed based on tagssuch as the chapter number and the line number. The tags may beinferred, or extracted or present in the underlying document. Users canembed quotes from documents. For example, a webpage may contain anembedded quote to a line from a chapter of a book. In an exemplaryembodiment, hovering over the quotation or clicking on the quotation maydisplay the corresponding quotation. In an exemplary embodiment,embedding a quotation tag with an identification number may display thequotation in the document in which the quotation is embedded. Quotationscan be used for text, audio, video, or other media. A version number maybe used for related documents. The system enables the user to findrelated quotes or verses. “Quotation chains” may also be supported.Quotation chains enable the user to quote a document that in turn quotesanother document so that the source of the information can be traced.

The system 10 has been described herein with regards to being accessibleonly through the Internet, where a server application is resident upon aserver 20. The respective applications that provide the functionalitiesthat have been described above, may be installed on a localizedstand-alone devices in alternative embodiments. The respective apparelitems and other products that the user may view and or selected, maythen be downloaded to the respective device upon connecting to anInternet server. The stand-alone devices in alternative embodiments maycommunicate with the server, where the server has access to variousdatabases and repositories wherein items and offerings may be stored.These stand-alone devices may be available as terminals or stations at astore, which may be linked to store inventories. Using these terminals,it may be possible to search via keywords, voice, image, barcode andspecify filters like price range.

While the above description provides examples of the embodiments, itwill be appreciated that some features and/or functions of the describedembodiments are susceptible to modification without departing from thespirit and principles of operation of the described embodiments.Furthermore, the systems, methods, features and/or functions describedabove may be used independently or in conjunction with other systemsand/or methods; and may be applied or used in other context other thanthe those mentioned in this document. Accordingly, what has beendescribed above has been intended to be illustrative of the inventionand non-limiting and it will be understood by persons skilled in the artthat other variants and modifications may be made without departing fromthe scope of the invention as defined in the claims appended hereto.

REFERENCES

-   [1] M. Desbrun, M. Meyer and P. Alliez, “Intrinsic Parameterizations    of Surface Meshes,” Comput. Graphics Forum, vol. 21, pp. 209-218,    2002.-   [2] H. Y. Bennamoun and M., “1D-PCA, 2D-PCA to nD-PCA,” Pattern    Recognition, 2006. ICPR 2006. 18th International Conference on, vol.    4, pp. 181-184, 2006.-   [3] R. Davis, “Magic Paper: Sketch-Understanding Research,”    Computer, vol. 40, pp. 34-41, 2007.-   [4] O. Bimber and R. Raskar, Spatial Augmented Reality: Merging Real    and Virtual Worlds. A K Peters, Ltd., 2005,-   [5] G. L. Congdong and Li, “Collaborative Filtering Recommendation    Model Through Active Bayesian Classifier,” Information Acquisition,    2006 IEEE International Conference on, pp. 572-577, August 2006.-   [6] T. Yoshioka and S. Ishii, “Fast Gaussian process regression    using representative data,” Neural Networks, 2001. Proceedings.    IJCNN '01. International Joint Conference on, vol. 1, pp. 132-137    0.1, 2001.-   [7] D. J. Hand and K. Yu, “Idiot's Bayes: Not So Stupid After All?”    International Statistical Review, vol. 69, pp. 385-398, 2001.-   [8] P. A. Flach and N. Lachiche, “Naive Bayesian Classification of    Structured Data,” Machine Learning, vol. 57, pp. 233-269,    December 2004. 2004.-   [9] T. Hastie, R. Tibshirani and J. H. Friedman, The Elements of    Statistical Learning. Springer, 2001,-   [10] G. Shakhnarovich, T. Darrell and P. Indyk, Nearest-Neighbor    Methods in Learning and Vision: Theory and Practice (Neural    Information Processing). The MIT Press, 2006,-   [11] B. Froba and C. Kubibeck, “Robust face detection at video frame    rate based on edge orientation features,” Automatic Face and Gesture    Recognition, 2002. Proceedings. Fifth IEEE International Conference    on, pp. 327-332, 20-21 May 2002.-   [12] A. R. Chowdhury, R. Chellappa, S. Krishnamurthy and T. Vo, “3D    face reconstruction from video using a generic model,” Multimedia    and Expo, 2002. ICME '02. Proceedings. 2002 IEEE International    Conference on, vol. 1, pp. 449-452 o. 1, 2002.-   [13] L. D. Alvarez, J. Mateos, R. Molina and A. K. Katsaggelos,    “High&hyphen; resolution images from compressed low&hyphen;    resolution video: Motion estimation and observable pixels,” Int. J.    Imaging Syst. Technol., vol. 14, pp. 58-66, 2004.2004.-   [14] U. P. Jain and Anil K., “3D Face Reconstruction from Stereo    Video,” Computer and Robot Vision, 2006. the 3rd Canadian Conference    on, pp. 41-41, 7-9 Jun. 2006.-   [15] H. Zhang, A. C. Berg, M. Maire and J. Malik, “SVM-KNN:    Discriminative nearest neighbor classification for visual category    recognition,” in CVPR '06: Proceedings of the 2006 IEEE Computer    Society Conference on Computer Vision and Pattern Recognition, 2006,    pp. 2126-2136.-   [16] V. Perlibakas, “Automatical detection of face features and    exact face contour,” Pattern Recognition Letters, vol. 24, pp.    2977-2985, December, 2003.2003.-   [17] P. Kuo and J. Hannah, “Improved Chin Fitting Algorithm Based on    An Adaptive Snake,” Image Processing, 2006 IEEE International    Conference on, pp. 205-208, 8-11 Oct. 2006.-   [18] M. Castel and E. R. Hancock, “Acquiring height data from a    single image of a face using local shape indicators,” Comput. Vis.    Image Underst., vol. 103, pp. 64-79, 2006.-   [19] P. L. Worthington, “Reillumination-driven shape from shading,”    Comput. Vis. Image Underst., vol. 98, pp. 326-344, 2005.-   [20] G. Vogiatzis, P. Favaro and R. Cipolla, “Using frontier points    to recover shape, reflectance and illumination,” Computer Vision,    2005. ICCV 2005. Tenth IEEE International Conference on, vol. 1, pp.    228-235 o. 1, 17-21 Oct. 2005.-   [21] C. H. Esteban and F. Schmitt, “Silhouette and stereo fusion for    3D object modeling,” Comput. Vis. Image Underst., vol. 96, pp.    367-392, 2004.-   [22] R. Dovgard and R. Basri, “Statistical symmetric shape from    shading for 3D structure recovery of faces,” Lecture Notes in    Computer Science, vol. 3022, pp. 99-113, 2004.-   [23] H. Murase and S. K. Nayar, “Visual learning and recognition of    3-D objects from appearance,” Int. J. Comput. Vision, vol. 14, pp.    5-24, 1995.-   [24] Y. Iwasaki, T. Kaneko and S. Kuriyama, “3D hair modeling based    on CT data and photographs.” in Computer Graphics and Imaging, 2003,    pp. 123-128.-   [25] Y. G. Zhiyong and Huang, “A method of human short hair modeling    and real time animation,” Computer Graphics and Applications, 2002.    Proceedings. 10th Pacific Conference on, pp. 435-438, 2002.-   [26] K. Ward, F. Bertails, TaeYong Kim, S. R. Marschner, M. P. Cani    and M. C. Lin, “A Survey on Hair Modeling: Styling, Simulation, and    Rendering,” Transactions on Visualization and Computer Graphics,    vol. 13, pp. 213-234, March-April 2007.-   [27] A. S. Micilotta, E. Ong and R. Bowden, “Real-time upper body    detection and 3D pose estimation in monoscopic images,” in    Proceedings of the European Conference on Computer Vision    (ECCV'06)-Volume 3; Lecture Notes in Computer Science, 2006, pp.    139-150.-   [28] L. Tong-Yee and H. Po-Hua, “Fast and intuitive metamorphosis of    3D polyhedral models using SMCC mesh merging scheme,” Visualization    and Computer Graphics, IEEE Transactions on, vol. 9, pp. 85-98,    2003.2003.-   [29] B. Allen, B. Curless and Z. Popovic, “The space of human body    shapes: Reconstruction and parameterization from range scans,” in    SIGGRAPH '03: ACM SIGGRAPH 2003 Papers, 2003, pp. 587-594.-   [30] V. Blanz and T. Vetter, “A morphable model for the synthesis of    3D faces,” in SIGGRAPH '99: Proceedings of the 26th Annual    Conference on Computer Graphics and Interactive Techniques, 1999,    pp. 187-194.-   [31] I. Baran and J. Popovic, “Automatic rigging and animation of 3D    characters,” ACM Trans. Graph., vol. 26, pp. 72, 2007.-   [32] A. Hilton, J. Starck and G. Collins, “From 3D Shape Capture to    Animated Models,” International Symposium on 3D Data Processing,    Visualization and Transmission (3DPVT), pp. 246-257, 2002.-   [33] X. Yang, A. Somasekharan and J. J. Zhang, “Curve skeleton    skinning for human and creature characters: Research Articles,”    Comput. Animat. Virtual Worlds, vol. 17, pp. 281-292, 2006.-   [34] W. T. Tang and ChiKeung, “Multiresolution Mesh Reconstruction    from Noisy 3D Point Sets,” Pattern Recognition, 2006. ICPR 2006.    18th International Conference on, vol. 1, pp. 5-8, 2006.-   [35] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers    and J. Davis, “SCAPE: shape completion and animation of people,” ACM    Trans. Graph., vol. 24, pp. 408-416, 2005.-   [36] F. L. Matthews and J. B. West, “Finite element displacement    analysis of a lung,” Journal of Biomechanics, vol. 5, pp. 591-600,    November, 1972. 1972.-   [37] S. H. Sundaram and C. C. Feng, “Finite element analysis of the    human thorax,” Journal of Biomechanics, vol. 10, pp. 505-516, 1977.    1977.-   [38] Y. Zhang, Y. Qiu, D. B. Goldgof, S. Sarkar and L. Li, “3D    finite element modeling of nonrigid breast deformation for feature    registration in-ray and MR images,” in WACV '07: Proceedings of the    Eighth IEEE Workshop on Applications of Computer Vision, 2007, pp.    38.-   [39] G. Sakas, L. Schreyer and M. Grimm, “Preprocessing and Volume    Rendering of 3D Ultrasonic Data,” IEEE Comput. Graph. Appl., vol.    15, pp. 47-54, 1995.-   [40] Y. Ito, P. Corey Shum, A. M. Shih, B. K. Soni and K. Nakahashi,    “Robust generation of high-quality unstructured meshes on realistic    biomedical geometry,” Int. J. Numer. Meth. Engng, vol. 65, pp.    943-973, 5 Feb. 2006. 2006.-   [41] S. Zhao and H. Lee, “Human silhouette extraction based on HMM,”    in ICPR '06: Proceedings of the 18th International Conference on    Pattern Recognition, 2006, pp. 994-997.-   [42] A. C. Kak and M. Slaney, Principles of Computerized Tomographic    Imaging. Philadelphia, Pa., USA: Society for Industrial and Applied    Mathematics, 2001,-   [43] S. Linnainmaa, D. Harwood and L. S. Davis, “Pose determination    of a three-dimensional object using triangle pairs,” Pattern    Analysis and Machine Intelligence, IEEE Transactions on, vol. 10,    pp. 634-647, September 1988. 1988.-   [44] D. Hoiem, A. A. Efros and M. Hebert, “Automatic photo pop-up,”    in SIGGRAPH '05: ACM SIGGRAPH 2005 Papers, 2005, pp. 577-584.-   [45] M. Desbrun, M. Meyer and P. Alliez, “Intrinsic    Parameterizations of Surface Meshes,” Computer Graphics Forum, vol.    21, pp. 209-218, September 2002. 2002.-   [46] M. C. Lincoln and A. F. Clark, “Pose-independent face    identification from video sequences,” in AVBPA '01: Proceedings of    the Third International Conference on Audio-and Video-Based    Biometric Person Authentication, 2001, pp. 14-19.-   [47] I. Sato, Y. Sato and K. Ikeuchi, “Illumination from shadows,”    Transactions on Pattern Analysis and Machine Intelligence, vol. 25,    pp. 290-300, March 2003.-   [48] P. Rheingans and D. Ebert, “Volume illustration:    nonphotorealistic rendering of volume models,” Visualization and    Computer Graphics, IEEE Transactions on, vol. 7, pp. 253-264,    July 2001. 2001.-   [49] A. Finkelstein and L. Markosian, “Nonphotorealistic rendering,”    IEEE Computer Graphics and Applications, vol. 23, pp. 26-27,    July 2003. 2003.-   [50] J. Lansdown and S. Schofield, “Expressive rendering: a review    of nonphotorealistic techniques,” IEEE Computer Graphics and    Applications, vol. 15, pp. 29-37, May 1995.1995.-   [51] J. M. Wang, D. J. Fleet and A. Hertzmann, “Multifactor Gaussian    process models for style-content separation,” in ICML '07:    Proceedings of the 24th International Conference on Machine    Learning, 2007, pp. 975-982.-   [52] M. D. Cordea, E. M. Petriu and D. C. Petriu, “3D Head Tracking    and Facial Expression Recovery using an Anthropometric Muscle-based    Active Appearance Model,” Instrumentation and Measurement Technology    Conference Proceedings, 2007 IEEE, pp. 1-6, 1-3 May 2007.-   [53] Z. Hammal, L. Couvreur, A. Caplier and M. Rombaut, “Facial    expression classification: An approach based on the fusion of facial    deformations using the transferable belief model,” International    Journal of Approximate Reasoning, vol. 46, pp. 542-567,    December, 2007. 2007.-   [54] J. X. Chen, Y. Yang and X. Wang, “Physics-Based Modeling and    Real-Time Simulation,” Computing in Science and Engg., vol. 3, pp.    98-102, 2001.-   [55] A. Shahhosseini and G. M. Knapp, “Semantic image retrieval    based on probabilistic latent semantic analysis,” in MULTIMEDIA '06:    Proceedings of the 14th Annual ACM International Conference on    Multimedia, 2006, pp. 703-706.-   [56] H. Snoussi and A. MohammadDjafari, “Bayesian Unsupervised    Learning for Source Separation with Mixture of Gaussians Prior,” J.    VLSI Signal Process. Syst., vol. 37, pp. 263-279, 2004.-   [57] Y. Zhang and C. Zhang, “Separation of music signals by harmonic    structure modeling,” in Advances in Neural Information Processing    Systems 18 Y. Weiss, B. Scholkopf and J. Platt, Eds. Cambridge,    Mass.: MIT Press, 2006, pp. 1617-1624.-   [58] Y. Yang, Y. Su, Y. Lin and H. H. Chen, “Music emotion    recognition: The role of individuality,” in HCM '07: Proceedings of    the International Workshop on Human-Centered Multimedia, 2007, pp.    13-22.-   [59] Tao Li and M. Ogihara, “Content-based music similarity search    and emotion detection,” Acoustics, Speech, and Signal Processing,    2004. Proceedings. (ICASSP '04). IEEE International Conference on,    vol. 5, pp. -705-8 o. 5, 17-21 May 2004.-   [60] X. Zhu, A. B. Goldberg, M. Eldawy, C. R. Dyer and B. Strock, “A    Text-to-Picture Synthesis System for Augmenting Communication.” pp.    1590, 2007.-   [61] D. D. Lewis, “Naive (Bayes) at forty: The independence    assumption in information retrieval,” in ECML '98: Proceedings of    the 10th European Conference on Machine Learning, 1998, pp. 4-15.-   [62] H. Zhang and J. Su, “Naive bayesian classifiers for ranking,”    in ECML, 2004, pp. 501-512.-   [63] D. A. Vasquez Govea and T. Fraichard, “Motion Prediction for    Moving Objects: a Statistical Approach,” pp. 3931-3936, April. 2004.-   [64] B. K. P. Horn and B. G. Schunck, “Determining optical flow,”    Massachusetts Institute of Technology, Cambridge, Mass., USA, 1980.

1. A method of sharing the amount of an online purchase transactioncomprising the steps of: a) initiating a transaction between one or moreusers from among a plurality of users; b) selecting one or more of theusers participating in the transaction; c) allocating the amount of thetransaction between some or all of the selected users; and d) completingthe transaction if each of the selected users pays the portion of theamount distributed to the selected user and if the total of amounts paidby the users matches the amount of the transaction.
 2. A method as inclaim 1 wherein if any of the selected users does not pay the amount ofthe transaction allocated to that user within a specified time, thetransaction is declined, and: a) releasing any hold placed on amountsauthorized for payment by a selected user; and b) refunding any amountactually paid by a selected user.
 3. A method as in claim 1 whereininformation may be shown to the users as they propose an allocation ofthe amount between users, including the portion of the amount remainingto be allocated and optionally the taxes and tip corresponding to eachuser.
 4. A method as in claim 1 wherein the users can chose to apply anyarbitrary allocation between users including allocating the amountevenly between users; each user paying for his/her items; or one or moreusers paying for all of the users.
 5. A method as in claim 1 wherein theusers can be online or offline. If the user(s) is offline, then thatuser(s) is sent a notification to share the transaction via one or moremeans including a notification on the hosting website or on any othersocial networking websites like Facebook, MySpace, and Friendster, or ona chat application such as MSN chat or via email or on a cell phonecommunicating via text such as through SMS or via voice by employingtext to speech conversion. Users can also schedule a time to be onlinevia these means and via a shared calendar.
 6. A method as in claim 1wherein users can categorize portions of the amount for various purposesincluding claiming reimbursement for expenses incurred on the part of anemployee for the purposes of work.
 7. A method as in claim 1 wherein acopy of the transaction is saved as an electronic receipt for purposesincluding returns or exchanges.
 8. A method as in claim 1 that isfurther extended to include point of sale terminals.
 9. A method ofcollaborative online shopping comprising: a) browsing and shopping withother users in shared collaborative environments including web browsersand applications simulating virtual mall and store environments; b)selective sharing of account resources with selected users, whereaccount resources may include users' current views (the contentcurrently being viewed by a user), shopping carts, shopping lists,wishlists, fitting rooms, user models, products of interest to the user,messages, bills, audio and other multimedia files and play lists,images, and other multimedia content, users' ratings, feedback,user-specified content regarding a product including comments and tags;and c) communication and interaction between users via means includingvoice, chat, text and other online and electronic communication means,while shopping.
 10. A method as in claim 8 wherein the mode ofinteraction is asynchronous, in which collaboration including browsing,shopping, sharing, communication and interaction can be performedwithout requiring other collaborators to be online.
 11. A method as inclaim 8 wherein the mode of interaction is synchronous, in whichcollaborators are online and synchronized collaboration includingbrowsing, shopping, sharing, communication and interaction is performed.12. A method as in claim 8 wherein the mode of interaction is common, inwhich collaborators are simultaneously engaged in synchronizedcollaboration, including browsing, shopping, sharing, communication andinteraction, in a common environment.
 13. A method as in claim 8 whereinusers can collaborate with friends on social networks.
 14. A method asin claim 8 where tools and assistance are provided by the system tofacilitate collaborative activities between users that take into accountgroup preferences and needs. Instances of this include: a) A tool forscheduling a time to go on a collaborative trip online. b) The systemcan also propose locations for group activities including a location ofa place of interest that minimizes the travel for all the users in thecollaborative session. c) Facility for users to organize event,activity, or occasion information and description for any activity orevent which may involve a group including, but is not limited to,details such as the theme, location, venue, map information,participants, attendees, dress code, news, feeds and articles related tothe event, photos, videos and other event related media, user feedback,ratings and comments, which can be posted and viewed. Users can shareviews of themselves (either their image, photo, video, other media ortheir 3D character model) in celebrity or movie apparel, or the apparelthey plan to wear to a particular event, activity, or occasion to sharethe spirit of the occasion which they plan to attend. d) Suggestions onwhat to wear for a particular occasion, event, or activity, and what tobring to an event, activity, or occasion and where to buy it can beprovided by the system taking into account and processing userpreferences and event, activity, or occasion details. Apparel and venuedecorations suggestions for the event, activity, or occasion areprovided based on the season, time of day the event or activity is held,whether the event is indoor or outdoor, and budget allocated. Otherevent-specific factors may be taken into account to assist incoordinating apparel to be worn by collaborating users who are going toan event. e) Information on restaurants, shopping plazas, movie studios,games, historical sites, museums and other venues; upcoming events andshows, festivals, concerts, and exhibitions, and music bands/groups,celebrities coming to town is made available and suggestions on where togo are provided by the system. The system may also incorporate users'preferences, and/or proximity of the event and other user-specific orsystem default criteria to make suggestions. Users may also obtain thelatest weather and traffic updates as well as all traffic and weatherinformation relevant to a given event, venue, or activity. f) Users cancollaboratively design a room or any space virtually and purchasevirtual furniture, or design, build and buy furniture or other items andthe corresponding real furnishings and decorations to furnish thecorresponding real space.
 15. A method of product recommendationcomprising: a) collecting personal user data including profession,gender, size, preferences, user's apparel size, user's address, who theuser's friend are, user's friends' information, users' medical recordsincluding eyeglass and contact lens prescription information; b)collecting vendor data including product size, product description,product location, price; and c) recommending vendor products that bestmatch the users' personal data.
 16. A method as in claim 15 wherein thesaid user and vendor data are stored for future reference andrecommendation.
 17. A method as in claim 15 wherein users can shop forand buy products for their friends that are compatible with theirfriend's personal information including apparel that fits them, withoutcompromising their friend's privacy.
 18. A method as in claim 15 inwhich accurate 3D body models representing the user are generated,comprising: a) acquisition of multimedia from the user for extraction ofdata pertaining to physical attributes; b) controls for dynamicallyadjusting dimensions of various body parts of the model; c) use offeedback provided by the user on body information; d) combining of 2Duser images and anthropometric data to construct a 3D body and facemodel of the user; e) applying optimization techniques to the generatedmodel to increase precision of match with the user's real face and body;f) refining the 3D model using texture maps, pattern, color, shape andother information pertaining to the make and material of the apparel toprovide photorealism; g) creating custom looks on the 3D model byselecting apparel, cosmetic, hair and dental products from catalogues orby performing a virtual makeover.
 19. A method as in claim 17 whereinreal-time goodness of fit information is communicated to the user as theuser browses through apparel. The goodness of fit information includesinformation in the form of: a) Flashing arrows or varied color regionsand/or other graphic or visual indicator, for instance, to indicate typeof fit (tight, loose and other degrees of fit) in a region and whereadjustments need to be made for proper fitting; b) Providing the userwith a close up view, zooming onto a specific area of interest on the 3Dmodel to view and examine fit in that region; c) Using a tape measureanimation to indicate the dimensions of a particular body segment orregion; d) Digital effects such as a transparency/x-ray vision effectwhere the apparel's transparency can be changed in order to enable theuser to examine fit in the particular region; e) Specifying numericmeasurements to indicate fit information including the gap or marginbetween apparel and body in different regions, after apparel is worn; anoverall goodness of fit rating.
 20. A method as in claim 17 whereinproducts that are relevant to the user's personal data are shown to theuser as the user browses through products including apparel that fitsthe user, products that matches the user's medical records includingeyeglasses or contact lenses that match the user's prescription.